2023-02-13 00:53:35,678 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart timeout: 13 mins 2023-02-13 00:53:35,682 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestartSplitWithoutZk timeout: 13 mins 2023-02-13 00:53:35,726 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=7, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=326, ProcessCount=168, AvailableMemoryMB=5319 2023-02-13 00:53:35,970 DEBUG [Time-limited test] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218 2023-02-13 00:53:35,971 INFO [Time-limited test] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-13 00:53:35,975 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 00:53:35,975 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50, deleteOnExit=true 2023-02-13 00:53:35,975 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 00:53:35,976 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/test.cache.data in system properties and HBase conf 2023-02-13 00:53:35,976 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 00:53:35,977 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/hadoop.log.dir in system properties and HBase conf 2023-02-13 00:53:35,977 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 00:53:35,977 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 00:53:35,978 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 00:53:36,081 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-13 00:53:36,506 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 00:53:36,512 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 00:53:36,512 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 00:53:36,513 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 00:53:36,513 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 00:53:36,514 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 00:53:36,514 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 00:53:36,515 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 00:53:36,515 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 00:53:36,516 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 00:53:36,516 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/nfs.dump.dir in system properties and HBase conf 2023-02-13 00:53:36,517 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/java.io.tmpdir in system properties and HBase conf 2023-02-13 00:53:36,517 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 00:53:36,517 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 00:53:36,518 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 00:53:37,076 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 00:53:37,080 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 00:53:37,399 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-13 00:53:37,584 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-13 00:53:37,604 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 00:53:37,645 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 00:53:37,678 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/java.io.tmpdir/Jetty_localhost_39389_hdfs____.vft4aw/webapp 2023-02-13 00:53:37,851 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39389 2023-02-13 00:53:37,877 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 00:53:37,878 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 00:53:38,376 WARN [Listener at localhost/45541] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 00:53:38,491 WARN [Listener at localhost/45541] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 00:53:38,512 WARN [Listener at localhost/45541] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 00:53:38,520 INFO [Listener at localhost/45541] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 00:53:38,524 INFO [Listener at localhost/45541] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/java.io.tmpdir/Jetty_localhost_42491_datanode____fvonp1/webapp 2023-02-13 00:53:38,634 INFO [Listener at localhost/45541] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42491 2023-02-13 00:53:39,017 WARN [Listener at localhost/39373] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 00:53:39,040 WARN [Listener at localhost/39373] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 00:53:39,045 WARN [Listener at localhost/39373] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 00:53:39,047 INFO [Listener at localhost/39373] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 00:53:39,055 INFO [Listener at localhost/39373] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/java.io.tmpdir/Jetty_localhost_33325_datanode____.rbhz7/webapp 2023-02-13 00:53:39,200 INFO [Listener at localhost/39373] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33325 2023-02-13 00:53:39,217 WARN [Listener at localhost/38635] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 00:53:39,265 WARN [Listener at localhost/38635] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 00:53:39,275 WARN [Listener at localhost/38635] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 00:53:39,278 INFO [Listener at localhost/38635] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 00:53:39,293 INFO [Listener at localhost/38635] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/java.io.tmpdir/Jetty_localhost_33149_datanode____.fq1wqr/webapp 2023-02-13 00:53:39,433 INFO [Listener at localhost/38635] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33149 2023-02-13 00:53:39,455 WARN [Listener at localhost/37931] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 00:53:39,645 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x66fe5f6d0fd02db6: Processing first storage report for DS-012f2b2e-6845-4514-b01e-82821451c519 from datanode 60127b70-1772-4fff-80b6-3be9184b83aa 2023-02-13 00:53:39,647 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x66fe5f6d0fd02db6: from storage DS-012f2b2e-6845-4514-b01e-82821451c519 node DatanodeRegistration(127.0.0.1:43395, datanodeUuid=60127b70-1772-4fff-80b6-3be9184b83aa, infoPort=36553, infoSecurePort=0, ipcPort=39373, storageInfo=lv=-57;cid=testClusterID;nsid=1442559003;c=1676249617163), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-13 00:53:39,647 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe5a3af225f0d1a7b: Processing first storage report for DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e from datanode f126e0b0-7796-4bf2-8e9f-858da0a2bce8 2023-02-13 00:53:39,647 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe5a3af225f0d1a7b: from storage DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e node DatanodeRegistration(127.0.0.1:41349, datanodeUuid=f126e0b0-7796-4bf2-8e9f-858da0a2bce8, infoPort=34949, infoSecurePort=0, ipcPort=38635, storageInfo=lv=-57;cid=testClusterID;nsid=1442559003;c=1676249617163), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:39,647 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x66fe5f6d0fd02db6: Processing first storage report for DS-c70b5a9b-2e2b-4306-ba64-f231be93ae45 from datanode 60127b70-1772-4fff-80b6-3be9184b83aa 2023-02-13 00:53:39,648 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x66fe5f6d0fd02db6: from storage DS-c70b5a9b-2e2b-4306-ba64-f231be93ae45 node DatanodeRegistration(127.0.0.1:43395, datanodeUuid=60127b70-1772-4fff-80b6-3be9184b83aa, infoPort=36553, infoSecurePort=0, ipcPort=39373, storageInfo=lv=-57;cid=testClusterID;nsid=1442559003;c=1676249617163), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:39,648 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe5a3af225f0d1a7b: Processing first storage report for DS-b108d037-8ff5-4750-beb5-78e82340ad53 from datanode f126e0b0-7796-4bf2-8e9f-858da0a2bce8 2023-02-13 00:53:39,648 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe5a3af225f0d1a7b: from storage DS-b108d037-8ff5-4750-beb5-78e82340ad53 node DatanodeRegistration(127.0.0.1:41349, datanodeUuid=f126e0b0-7796-4bf2-8e9f-858da0a2bce8, infoPort=34949, infoSecurePort=0, ipcPort=38635, storageInfo=lv=-57;cid=testClusterID;nsid=1442559003;c=1676249617163), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:40,804 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7f475cb1b7ecda89: Processing first storage report for DS-8d798703-a000-4606-9d1c-5bb60d2f414a from datanode 569dd8da-56b0-4966-9330-63ec496574b2 2023-02-13 00:53:40,804 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7f475cb1b7ecda89: from storage DS-8d798703-a000-4606-9d1c-5bb60d2f414a node DatanodeRegistration(127.0.0.1:39413, datanodeUuid=569dd8da-56b0-4966-9330-63ec496574b2, infoPort=33133, infoSecurePort=0, ipcPort=37931, storageInfo=lv=-57;cid=testClusterID;nsid=1442559003;c=1676249617163), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:40,804 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7f475cb1b7ecda89: Processing first storage report for DS-ba6b26af-52b2-4d28-8700-d9a4073cb618 from datanode 569dd8da-56b0-4966-9330-63ec496574b2 2023-02-13 00:53:40,804 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7f475cb1b7ecda89: from storage DS-ba6b26af-52b2-4d28-8700-d9a4073cb618 node DatanodeRegistration(127.0.0.1:39413, datanodeUuid=569dd8da-56b0-4966-9330-63ec496574b2, infoPort=33133, infoSecurePort=0, ipcPort=37931, storageInfo=lv=-57;cid=testClusterID;nsid=1442559003;c=1676249617163), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:40,897 DEBUG [Listener at localhost/37931] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218 2023-02-13 00:53:40,958 INFO [Listener at localhost/37931] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50/zookeeper_0, clientPort=51121, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 00:53:40,971 INFO [Listener at localhost/37931] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51121 2023-02-13 00:53:40,981 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:40,984 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:41,676 INFO [Listener at localhost/37931] util.FSUtils(479): Created version file at hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc with version=8 2023-02-13 00:53:41,676 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/hbase-staging 2023-02-13 00:53:42,000 INFO [Listener at localhost/37931] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 00:53:42,466 INFO [Listener at localhost/37931] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:53:42,498 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:42,499 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:42,499 INFO [Listener at localhost/37931] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:42,499 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:42,499 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:42,655 INFO [Listener at localhost/37931] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:42,734 DEBUG [Listener at localhost/37931] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-13 00:53:42,828 INFO [Listener at localhost/37931] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35917 2023-02-13 00:53:42,839 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:42,841 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:42,863 INFO [Listener at localhost/37931] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35917 connecting to ZooKeeper ensemble=127.0.0.1:51121 2023-02-13 00:53:42,909 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:359170x0, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:42,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:35917-0x1004b2b4b940000 connected 2023-02-13 00:53:42,941 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:42,942 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:42,947 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:42,958 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35917 2023-02-13 00:53:42,959 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35917 2023-02-13 00:53:42,960 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35917 2023-02-13 00:53:42,961 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35917 2023-02-13 00:53:42,962 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35917 2023-02-13 00:53:42,970 INFO [Listener at localhost/37931] master.HMaster(439): hbase.rootdir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc, hbase.cluster.distributed=false 2023-02-13 00:53:43,052 INFO [Listener at localhost/37931] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:53:43,052 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:43,052 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:43,052 INFO [Listener at localhost/37931] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:43,052 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:43,053 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:43,058 INFO [Listener at localhost/37931] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:43,061 INFO [Listener at localhost/37931] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46237 2023-02-13 00:53:43,063 INFO [Listener at localhost/37931] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:53:43,070 DEBUG [Listener at localhost/37931] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:53:43,071 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:43,073 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:43,076 INFO [Listener at localhost/37931] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46237 connecting to ZooKeeper ensemble=127.0.0.1:51121 2023-02-13 00:53:43,083 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:462370x0, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:43,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46237-0x1004b2b4b940001 connected 2023-02-13 00:53:43,084 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:43,086 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:43,087 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:43,088 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46237 2023-02-13 00:53:43,089 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46237 2023-02-13 00:53:43,091 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46237 2023-02-13 00:53:43,092 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46237 2023-02-13 00:53:43,092 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46237 2023-02-13 00:53:43,105 INFO [Listener at localhost/37931] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:53:43,105 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:43,106 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:43,107 INFO [Listener at localhost/37931] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:43,107 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:43,107 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:43,107 INFO [Listener at localhost/37931] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:43,109 INFO [Listener at localhost/37931] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36081 2023-02-13 00:53:43,110 INFO [Listener at localhost/37931] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:53:43,112 DEBUG [Listener at localhost/37931] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:53:43,113 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:43,116 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:43,118 INFO [Listener at localhost/37931] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36081 connecting to ZooKeeper ensemble=127.0.0.1:51121 2023-02-13 00:53:43,123 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:360810x0, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:43,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36081-0x1004b2b4b940002 connected 2023-02-13 00:53:43,125 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:43,125 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:43,126 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:43,127 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36081 2023-02-13 00:53:43,127 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36081 2023-02-13 00:53:43,127 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36081 2023-02-13 00:53:43,131 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36081 2023-02-13 00:53:43,131 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36081 2023-02-13 00:53:43,144 INFO [Listener at localhost/37931] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:53:43,144 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:43,144 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:43,144 INFO [Listener at localhost/37931] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:43,144 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:43,144 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:43,145 INFO [Listener at localhost/37931] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:43,146 INFO [Listener at localhost/37931] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38169 2023-02-13 00:53:43,147 INFO [Listener at localhost/37931] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:53:43,148 DEBUG [Listener at localhost/37931] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:53:43,149 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:43,151 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:43,152 INFO [Listener at localhost/37931] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38169 connecting to ZooKeeper ensemble=127.0.0.1:51121 2023-02-13 00:53:43,155 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:381690x0, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:43,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38169-0x1004b2b4b940003 connected 2023-02-13 00:53:43,156 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:43,157 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:43,158 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:43,160 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38169 2023-02-13 00:53:43,161 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38169 2023-02-13 00:53:43,163 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38169 2023-02-13 00:53:43,164 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38169 2023-02-13 00:53:43,168 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38169 2023-02-13 00:53:43,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:43,178 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 00:53:43,179 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:43,198 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:43,198 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:43,198 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:43,198 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:43,199 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:43,201 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:53:43,203 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,35917,1676249621835 from backup master directory 2023-02-13 00:53:43,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:53:43,207 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:43,207 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 00:53:43,207 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:43,207 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:43,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-13 00:53:43,213 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-13 00:53:43,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/hbase.id with ID: a53384a2-dda6-4252-aa83-796fb497e498 2023-02-13 00:53:43,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:43,371 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:43,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0fa48e27 to 127.0.0.1:51121 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:43,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@399ae35c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:43,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:43,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 00:53:43,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-13 00:53:43,526 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-13 00:53:43,527 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:53:43,531 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:53:43,533 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:43,566 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store-tmp 2023-02-13 00:53:43,602 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:43,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 00:53:43,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:43,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:43,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 00:53:43,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:43,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:43,604 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676249623602Disabling compacts and flushes for region at 1676249623603 (+1 ms)Disabling writes for close at 1676249623603Writing region close event to WAL at 1676249623603Closed at 1676249623603 2023-02-13 00:53:43,605 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:43,632 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35917%2C1676249621835, suffix=, logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,35917,1676249621835, archiveDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/oldWALs, maxLogs=10 2023-02-13 00:53:43,692 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK] 2023-02-13 00:53:43,692 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK] 2023-02-13 00:53:43,692 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK] 2023-02-13 00:53:43,700 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:53:43,767 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,35917,1676249621835/jenkins-hbase4.apache.org%2C35917%2C1676249621835.1676249623645 2023-02-13 00:53:43,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK], DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK], DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK]] 2023-02-13 00:53:43,768 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:43,768 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:43,771 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:43,772 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:43,839 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:43,847 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 00:53:43,879 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 00:53:43,893 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:43,899 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:43,900 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:43,915 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:43,919 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:43,921 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75341929, jitterRate=0.12268222868442535}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:43,921 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676249623797Initializing all the Stores at 1676249623799 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249623800 (+1 ms)Cleaning up temporary data from old regions at 1676249623906 (+106 ms)Cleaning up detritus from prior splits at 1676249623907 (+1 ms)Region opened successfully at 1676249623921 (+14 ms) 2023-02-13 00:53:43,922 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 00:53:43,946 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 00:53:43,946 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 00:53:43,948 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 00:53:43,950 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-13 00:53:43,981 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-13 00:53:43,981 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 00:53:44,005 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 00:53:44,011 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 00:53:44,040 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 00:53:44,045 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 00:53:44,046 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 00:53:44,052 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 00:53:44,055 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 00:53:44,059 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:44,060 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 00:53:44,061 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 00:53:44,072 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 00:53:44,076 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:44,076 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:44,076 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:44,076 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:44,076 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:44,077 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,35917,1676249621835, sessionid=0x1004b2b4b940000, setting cluster-up flag (Was=false) 2023-02-13 00:53:44,091 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:44,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 00:53:44,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:44,104 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:44,109 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 00:53:44,110 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:44,112 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.hbase-snapshot/.tmp 2023-02-13 00:53:44,172 INFO [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(952): ClusterId : a53384a2-dda6-4252-aa83-796fb497e498 2023-02-13 00:53:44,172 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(952): ClusterId : a53384a2-dda6-4252-aa83-796fb497e498 2023-02-13 00:53:44,172 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(952): ClusterId : a53384a2-dda6-4252-aa83-796fb497e498 2023-02-13 00:53:44,178 DEBUG [RS:2;jenkins-hbase4:38169] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:53:44,178 DEBUG [RS:0;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:53:44,178 DEBUG [RS:1;jenkins-hbase4:36081] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:53:44,185 DEBUG [RS:0;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:53:44,185 DEBUG [RS:2;jenkins-hbase4:38169] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:53:44,185 DEBUG [RS:1;jenkins-hbase4:36081] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:53:44,186 DEBUG [RS:2;jenkins-hbase4:38169] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:53:44,186 DEBUG [RS:0;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:53:44,186 DEBUG [RS:1;jenkins-hbase4:36081] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:53:44,190 DEBUG [RS:0;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:53:44,190 DEBUG [RS:2;jenkins-hbase4:38169] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:53:44,190 DEBUG [RS:1;jenkins-hbase4:36081] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:53:44,192 DEBUG [RS:0;jenkins-hbase4:46237] zookeeper.ReadOnlyZKClient(139): Connect 0x4fc6fdf1 to 127.0.0.1:51121 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:44,192 DEBUG [RS:1;jenkins-hbase4:36081] zookeeper.ReadOnlyZKClient(139): Connect 0x4eabd27d to 127.0.0.1:51121 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:44,192 DEBUG [RS:2;jenkins-hbase4:38169] zookeeper.ReadOnlyZKClient(139): Connect 0x563f9214 to 127.0.0.1:51121 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:44,235 DEBUG [RS:1;jenkins-hbase4:36081] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@406c25d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:44,236 DEBUG [RS:2;jenkins-hbase4:38169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58f866a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:44,236 DEBUG [RS:0;jenkins-hbase4:46237] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41011a79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:44,236 DEBUG [RS:2;jenkins-hbase4:38169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5cc7f964, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:44,236 DEBUG [RS:1;jenkins-hbase4:36081] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4453b90d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:44,236 DEBUG [RS:0;jenkins-hbase4:46237] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@453e9681, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:44,261 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 00:53:44,261 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:38169 2023-02-13 00:53:44,262 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:46237 2023-02-13 00:53:44,262 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:36081 2023-02-13 00:53:44,267 INFO [RS:2;jenkins-hbase4:38169] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:53:44,267 INFO [RS:0;jenkins-hbase4:46237] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:53:44,268 INFO [RS:0;jenkins-hbase4:46237] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:53:44,267 INFO [RS:1;jenkins-hbase4:36081] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:53:44,268 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:53:44,268 INFO [RS:2;jenkins-hbase4:38169] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:53:44,268 INFO [RS:1;jenkins-hbase4:36081] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:53:44,268 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:53:44,268 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:53:44,271 INFO [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35917,1676249621835 with isa=jenkins-hbase4.apache.org/172.31.14.131:38169, startcode=1676249623143 2023-02-13 00:53:44,271 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35917,1676249621835 with isa=jenkins-hbase4.apache.org/172.31.14.131:46237, startcode=1676249623051 2023-02-13 00:53:44,271 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35917,1676249621835 with isa=jenkins-hbase4.apache.org/172.31.14.131:36081, startcode=1676249623105 2023-02-13 00:53:44,272 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:44,272 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:44,272 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:44,273 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:44,273 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 00:53:44,273 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,273 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:44,273 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,278 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676249654278 2023-02-13 00:53:44,281 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 00:53:44,281 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 00:53:44,282 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 00:53:44,287 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:44,291 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 00:53:44,292 DEBUG [RS:2;jenkins-hbase4:38169] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:53:44,292 DEBUG [RS:1;jenkins-hbase4:36081] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:53:44,292 DEBUG [RS:0;jenkins-hbase4:46237] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:53:44,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 00:53:44,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 00:53:44,301 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 00:53:44,301 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 00:53:44,301 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 00:53:44,304 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 00:53:44,304 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 00:53:44,307 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 00:53:44,307 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 00:53:44,311 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249624309,5,FailOnTimeoutGroup] 2023-02-13 00:53:44,312 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249624311,5,FailOnTimeoutGroup] 2023-02-13 00:53:44,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 00:53:44,314 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,314 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,338 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:44,339 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:44,339 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc 2023-02-13 00:53:44,371 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49397, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:53:44,371 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41899, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:53:44,371 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43513, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:53:44,384 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:44,387 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35917] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:44,387 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35917] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:44,388 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 00:53:44,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:44,391 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info 2023-02-13 00:53:44,392 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 00:53:44,393 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:44,393 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 00:53:44,395 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:53:44,396 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 00:53:44,397 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:44,397 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 00:53:44,400 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/table 2023-02-13 00:53:44,401 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 00:53:44,405 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:44,407 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc 2023-02-13 00:53:44,408 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc 2023-02-13 00:53:44,408 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc 2023-02-13 00:53:44,408 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740 2023-02-13 00:53:44,409 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45541 2023-02-13 00:53:44,409 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45541 2023-02-13 00:53:44,408 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45541 2023-02-13 00:53:44,409 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:53:44,409 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:53:44,409 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:53:44,413 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740 2023-02-13 00:53:44,417 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 00:53:44,418 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:44,419 DEBUG [RS:2;jenkins-hbase4:38169] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:44,419 DEBUG [RS:0;jenkins-hbase4:46237] zookeeper.ZKUtil(162): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:44,419 DEBUG [RS:1;jenkins-hbase4:36081] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:44,419 WARN [RS:0;jenkins-hbase4:46237] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:44,420 INFO [RS:0;jenkins-hbase4:46237] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:44,419 WARN [RS:2;jenkins-hbase4:38169] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:44,420 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:44,420 INFO [RS:2;jenkins-hbase4:38169] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:44,421 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 00:53:44,421 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:44,421 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36081,1676249623105] 2023-02-13 00:53:44,421 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46237,1676249623051] 2023-02-13 00:53:44,420 WARN [RS:1;jenkins-hbase4:36081] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:44,421 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38169,1676249623143] 2023-02-13 00:53:44,421 INFO [RS:1;jenkins-hbase4:36081] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:44,421 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:44,433 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:44,434 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63330164, jitterRate=-0.05630701780319214}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 00:53:44,434 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676249624384Initializing all the Stores at 1676249624386 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249624386Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249624387 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249624387Cleaning up temporary data from old regions at 1676249624415 (+28 ms)Cleaning up detritus from prior splits at 1676249624416 (+1 ms)Region opened successfully at 1676249624434 (+18 ms) 2023-02-13 00:53:44,435 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 00:53:44,435 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 00:53:44,435 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 00:53:44,435 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 00:53:44,435 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 00:53:44,436 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 00:53:44,436 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676249624435Disabling compacts and flushes for region at 1676249624435Disabling writes for close at 1676249624435Writing region close event to WAL at 1676249624436 (+1 ms)Closed at 1676249624436 2023-02-13 00:53:44,439 DEBUG [RS:1;jenkins-hbase4:36081] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:44,439 DEBUG [RS:2;jenkins-hbase4:38169] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:44,440 DEBUG [RS:1;jenkins-hbase4:36081] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:44,441 DEBUG [RS:2;jenkins-hbase4:38169] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:44,441 DEBUG [RS:1;jenkins-hbase4:36081] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:44,442 DEBUG [RS:2;jenkins-hbase4:38169] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:44,442 DEBUG [RS:0;jenkins-hbase4:46237] zookeeper.ZKUtil(162): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:44,442 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 00:53:44,443 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 00:53:44,443 DEBUG [RS:0;jenkins-hbase4:46237] zookeeper.ZKUtil(162): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:44,446 DEBUG [RS:0;jenkins-hbase4:46237] zookeeper.ZKUtil(162): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:44,452 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 00:53:44,453 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:53:44,453 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:53:44,453 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:53:44,463 INFO [RS:0;jenkins-hbase4:46237] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:53:44,463 INFO [RS:1;jenkins-hbase4:36081] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:53:44,463 INFO [RS:2;jenkins-hbase4:38169] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:53:44,467 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 00:53:44,470 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 00:53:44,487 INFO [RS:2;jenkins-hbase4:38169] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:53:44,487 INFO [RS:0;jenkins-hbase4:46237] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:53:44,487 INFO [RS:1;jenkins-hbase4:36081] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:53:44,491 INFO [RS:1;jenkins-hbase4:36081] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:53:44,491 INFO [RS:0;jenkins-hbase4:46237] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:53:44,492 INFO [RS:1;jenkins-hbase4:36081] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,491 INFO [RS:2;jenkins-hbase4:38169] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:53:44,492 INFO [RS:0;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,492 INFO [RS:2;jenkins-hbase4:38169] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,493 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:53:44,493 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:53:44,493 INFO [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:53:44,501 INFO [RS:1;jenkins-hbase4:36081] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,501 INFO [RS:0;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,501 INFO [RS:2;jenkins-hbase4:38169] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,501 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,501 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,501 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,501 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,501 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:44,502 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:44,502 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,501 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:0;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,502 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,503 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,503 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,503 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:44,503 DEBUG [RS:1;jenkins-hbase4:36081] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,503 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,503 INFO [RS:0;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,503 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,504 INFO [RS:0;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,504 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,504 INFO [RS:0;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,504 DEBUG [RS:2;jenkins-hbase4:38169] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:44,504 INFO [RS:1;jenkins-hbase4:36081] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,504 INFO [RS:1;jenkins-hbase4:36081] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,505 INFO [RS:1;jenkins-hbase4:36081] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,508 INFO [RS:2;jenkins-hbase4:38169] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,508 INFO [RS:2;jenkins-hbase4:38169] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,508 INFO [RS:2;jenkins-hbase4:38169] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,525 INFO [RS:1;jenkins-hbase4:36081] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:53:44,525 INFO [RS:2;jenkins-hbase4:38169] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:53:44,525 INFO [RS:0;jenkins-hbase4:46237] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:53:44,528 INFO [RS:0;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46237,1676249623051-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,528 INFO [RS:1;jenkins-hbase4:36081] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36081,1676249623105-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,528 INFO [RS:2;jenkins-hbase4:38169] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38169,1676249623143-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:44,551 INFO [RS:1;jenkins-hbase4:36081] regionserver.Replication(203): jenkins-hbase4.apache.org,36081,1676249623105 started 2023-02-13 00:53:44,551 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36081,1676249623105, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36081, sessionid=0x1004b2b4b940002 2023-02-13 00:53:44,552 INFO [RS:2;jenkins-hbase4:38169] regionserver.Replication(203): jenkins-hbase4.apache.org,38169,1676249623143 started 2023-02-13 00:53:44,552 INFO [RS:0;jenkins-hbase4:46237] regionserver.Replication(203): jenkins-hbase4.apache.org,46237,1676249623051 started 2023-02-13 00:53:44,552 INFO [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38169,1676249623143, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38169, sessionid=0x1004b2b4b940003 2023-02-13 00:53:44,552 DEBUG [RS:1;jenkins-hbase4:36081] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:53:44,552 DEBUG [RS:2;jenkins-hbase4:38169] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:53:44,552 DEBUG [RS:1;jenkins-hbase4:36081] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:44,552 DEBUG [RS:2;jenkins-hbase4:38169] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:44,552 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46237,1676249623051, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46237, sessionid=0x1004b2b4b940001 2023-02-13 00:53:44,553 DEBUG [RS:2;jenkins-hbase4:38169] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38169,1676249623143' 2023-02-13 00:53:44,553 DEBUG [RS:0;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:53:44,553 DEBUG [RS:0;jenkins-hbase4:46237] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:44,553 DEBUG [RS:1;jenkins-hbase4:36081] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36081,1676249623105' 2023-02-13 00:53:44,553 DEBUG [RS:0;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46237,1676249623051' 2023-02-13 00:53:44,553 DEBUG [RS:2;jenkins-hbase4:38169] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:53:44,553 DEBUG [RS:0;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:53:44,553 DEBUG [RS:1;jenkins-hbase4:36081] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:53:44,554 DEBUG [RS:0;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:53:44,554 DEBUG [RS:1;jenkins-hbase4:36081] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:53:44,554 DEBUG [RS:2;jenkins-hbase4:38169] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:53:44,555 DEBUG [RS:0;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:53:44,555 DEBUG [RS:2;jenkins-hbase4:38169] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:53:44,555 DEBUG [RS:1;jenkins-hbase4:36081] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:53:44,555 DEBUG [RS:2;jenkins-hbase4:38169] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:53:44,555 DEBUG [RS:0;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:53:44,555 DEBUG [RS:2;jenkins-hbase4:38169] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:44,555 DEBUG [RS:0;jenkins-hbase4:46237] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:44,555 DEBUG [RS:1;jenkins-hbase4:36081] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:53:44,556 DEBUG [RS:0;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46237,1676249623051' 2023-02-13 00:53:44,556 DEBUG [RS:0;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:53:44,556 DEBUG [RS:2;jenkins-hbase4:38169] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38169,1676249623143' 2023-02-13 00:53:44,556 DEBUG [RS:2;jenkins-hbase4:38169] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:53:44,556 DEBUG [RS:1;jenkins-hbase4:36081] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:44,556 DEBUG [RS:1;jenkins-hbase4:36081] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36081,1676249623105' 2023-02-13 00:53:44,557 DEBUG [RS:1;jenkins-hbase4:36081] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:53:44,557 DEBUG [RS:0;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:53:44,557 DEBUG [RS:2;jenkins-hbase4:38169] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:53:44,557 DEBUG [RS:1;jenkins-hbase4:36081] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:53:44,557 DEBUG [RS:0;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:53:44,557 DEBUG [RS:2;jenkins-hbase4:38169] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:53:44,557 INFO [RS:0;jenkins-hbase4:46237] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:53:44,557 INFO [RS:2;jenkins-hbase4:38169] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:53:44,557 INFO [RS:0;jenkins-hbase4:46237] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:53:44,558 INFO [RS:2;jenkins-hbase4:38169] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:53:44,558 DEBUG [RS:1;jenkins-hbase4:36081] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:53:44,558 INFO [RS:1;jenkins-hbase4:36081] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:53:44,558 INFO [RS:1;jenkins-hbase4:36081] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:53:44,622 DEBUG [jenkins-hbase4:35917] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 00:53:44,625 DEBUG [jenkins-hbase4:35917] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:44,631 DEBUG [jenkins-hbase4:35917] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:44,631 DEBUG [jenkins-hbase4:35917] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:44,631 DEBUG [jenkins-hbase4:35917] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:44,631 DEBUG [jenkins-hbase4:35917] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:44,634 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46237,1676249623051, state=OPENING 2023-02-13 00:53:44,641 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 00:53:44,642 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:44,643 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 00:53:44,645 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46237,1676249623051}] 2023-02-13 00:53:44,668 INFO [RS:2;jenkins-hbase4:38169] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38169%2C1676249623143, suffix=, logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,38169,1676249623143, archiveDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs, maxLogs=32 2023-02-13 00:53:44,668 INFO [RS:1;jenkins-hbase4:36081] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36081%2C1676249623105, suffix=, logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,36081,1676249623105, archiveDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs, maxLogs=32 2023-02-13 00:53:44,668 INFO [RS:0;jenkins-hbase4:46237] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46237%2C1676249623051, suffix=, logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051, archiveDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs, maxLogs=32 2023-02-13 00:53:44,692 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK] 2023-02-13 00:53:44,692 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK] 2023-02-13 00:53:44,692 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK] 2023-02-13 00:53:44,700 INFO [RS:0;jenkins-hbase4:46237] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051/jenkins-hbase4.apache.org%2C46237%2C1676249623051.1676249624672 2023-02-13 00:53:44,700 DEBUG [RS:0;jenkins-hbase4:46237] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK], DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK], DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK]] 2023-02-13 00:53:44,707 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK] 2023-02-13 00:53:44,707 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK] 2023-02-13 00:53:44,707 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK] 2023-02-13 00:53:44,713 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK] 2023-02-13 00:53:44,713 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK] 2023-02-13 00:53:44,715 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK] 2023-02-13 00:53:44,729 INFO [RS:2;jenkins-hbase4:38169] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,38169,1676249623143/jenkins-hbase4.apache.org%2C38169%2C1676249623143.1676249624672 2023-02-13 00:53:44,729 INFO [RS:1;jenkins-hbase4:36081] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,36081,1676249623105/jenkins-hbase4.apache.org%2C36081%2C1676249623105.1676249624672 2023-02-13 00:53:44,730 DEBUG [RS:2;jenkins-hbase4:38169] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK], DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK], DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK]] 2023-02-13 00:53:44,732 DEBUG [RS:1;jenkins-hbase4:36081] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK], DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK], DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK]] 2023-02-13 00:53:44,833 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:44,835 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 00:53:44,840 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33216, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 00:53:44,855 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 00:53:44,855 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:44,859 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46237%2C1676249623051.meta, suffix=.meta, logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051, archiveDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs, maxLogs=32 2023-02-13 00:53:44,882 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK] 2023-02-13 00:53:44,883 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK] 2023-02-13 00:53:44,885 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK] 2023-02-13 00:53:44,892 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051/jenkins-hbase4.apache.org%2C46237%2C1676249623051.meta.1676249624861.meta 2023-02-13 00:53:44,893 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK], DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK], DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK]] 2023-02-13 00:53:44,893 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:44,895 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 00:53:44,916 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 00:53:44,921 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 00:53:44,926 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 00:53:44,926 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:44,927 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 00:53:44,927 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 00:53:44,929 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 00:53:44,931 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info 2023-02-13 00:53:44,931 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info 2023-02-13 00:53:44,932 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 00:53:44,933 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:44,933 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 00:53:44,934 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:53:44,934 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:53:44,935 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 00:53:44,936 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:44,936 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 00:53:44,938 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/table 2023-02-13 00:53:44,938 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/table 2023-02-13 00:53:44,939 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 00:53:44,939 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:44,941 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740 2023-02-13 00:53:44,945 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740 2023-02-13 00:53:44,951 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 00:53:44,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 00:53:44,954 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65412188, jitterRate=-0.025282442569732666}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 00:53:44,954 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676249624927Writing region info on filesystem at 1676249624927Initializing all the Stores at 1676249624929 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249624929Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249624929Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249624929Cleaning up temporary data from old regions at 1676249624949 (+20 ms)Cleaning up detritus from prior splits at 1676249624949Running coprocessor post-open hooks at 1676249624954 (+5 ms)Region opened successfully at 1676249624954 2023-02-13 00:53:44,965 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676249624824 2023-02-13 00:53:44,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 00:53:44,985 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 00:53:44,986 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46237,1676249623051, state=OPEN 2023-02-13 00:53:44,990 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 00:53:44,991 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 00:53:44,997 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 00:53:44,997 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46237,1676249623051 in 345 msec 2023-02-13 00:53:45,002 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 00:53:45,003 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 546 msec 2023-02-13 00:53:45,010 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 852 msec 2023-02-13 00:53:45,010 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676249623193, completionTime=-1 2023-02-13 00:53:45,010 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 00:53:45,010 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 00:53:45,082 DEBUG [hconnection-0x34521ec1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 00:53:45,085 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33228, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 00:53:45,103 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 00:53:45,103 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676249685103 2023-02-13 00:53:45,103 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676249745103 2023-02-13 00:53:45,104 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 93 msec 2023-02-13 00:53:45,132 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35917,1676249621835-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:45,132 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35917,1676249621835-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:45,132 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35917,1676249621835-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:45,134 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:35917, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:45,135 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:45,145 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 00:53:45,160 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 00:53:45,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:45,175 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 00:53:45,178 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 00:53:45,181 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 00:53:45,206 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:45,208 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8 empty. 2023-02-13 00:53:45,209 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:45,209 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 00:53:45,246 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:45,248 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => c7ca26b901efc47292da4c15f8e352a8, NAME => 'hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp 2023-02-13 00:53:45,270 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:45,270 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing c7ca26b901efc47292da4c15f8e352a8, disabling compactions & flushes 2023-02-13 00:53:45,270 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:45,270 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:45,270 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. after waiting 0 ms 2023-02-13 00:53:45,270 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:45,270 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:45,270 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for c7ca26b901efc47292da4c15f8e352a8: Waiting for close lock at 1676249625270Disabling compacts and flushes for region at 1676249625270Disabling writes for close at 1676249625270Writing region close event to WAL at 1676249625270Closed at 1676249625270 2023-02-13 00:53:45,275 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 00:53:45,290 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249625278"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676249625278"}]},"ts":"1676249625278"} 2023-02-13 00:53:45,320 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 00:53:45,322 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 00:53:45,327 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249625323"}]},"ts":"1676249625323"} 2023-02-13 00:53:45,331 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 00:53:45,335 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:45,337 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:45,337 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:45,337 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:45,337 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:45,339 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c7ca26b901efc47292da4c15f8e352a8, ASSIGN}] 2023-02-13 00:53:45,342 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c7ca26b901efc47292da4c15f8e352a8, ASSIGN 2023-02-13 00:53:45,344 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c7ca26b901efc47292da4c15f8e352a8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46237,1676249623051; forceNewPlan=false, retain=false 2023-02-13 00:53:45,496 INFO [jenkins-hbase4:35917] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 00:53:45,497 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c7ca26b901efc47292da4c15f8e352a8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:45,497 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249625497"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249625497"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249625497"}]},"ts":"1676249625497"} 2023-02-13 00:53:45,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c7ca26b901efc47292da4c15f8e352a8, server=jenkins-hbase4.apache.org,46237,1676249623051}] 2023-02-13 00:53:45,668 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:45,668 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c7ca26b901efc47292da4c15f8e352a8, NAME => 'hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:45,669 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:45,670 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:45,670 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:45,670 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:45,672 INFO [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:45,674 DEBUG [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/info 2023-02-13 00:53:45,675 DEBUG [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/info 2023-02-13 00:53:45,675 INFO [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7ca26b901efc47292da4c15f8e352a8 columnFamilyName info 2023-02-13 00:53:45,676 INFO [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] regionserver.HStore(310): Store=c7ca26b901efc47292da4c15f8e352a8/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:45,678 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:45,679 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:45,683 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:45,688 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:45,689 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c7ca26b901efc47292da4c15f8e352a8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74821801, jitterRate=0.11493171751499176}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:45,690 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c7ca26b901efc47292da4c15f8e352a8: Running coprocessor pre-open hook at 1676249625670Writing region info on filesystem at 1676249625670Initializing all the Stores at 1676249625672 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249625672Cleaning up temporary data from old regions at 1676249625679 (+7 ms)Cleaning up detritus from prior splits at 1676249625680 (+1 ms)Running coprocessor post-open hooks at 1676249625690 (+10 ms)Region opened successfully at 1676249625690 2023-02-13 00:53:45,697 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8., pid=6, masterSystemTime=1676249625655 2023-02-13 00:53:45,702 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:45,702 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:45,704 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c7ca26b901efc47292da4c15f8e352a8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:45,705 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249625703"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249625703"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249625703"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249625703"}]},"ts":"1676249625703"} 2023-02-13 00:53:45,714 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 00:53:45,714 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c7ca26b901efc47292da4c15f8e352a8, server=jenkins-hbase4.apache.org,46237,1676249623051 in 208 msec 2023-02-13 00:53:45,718 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 00:53:45,720 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c7ca26b901efc47292da4c15f8e352a8, ASSIGN in 375 msec 2023-02-13 00:53:45,721 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 00:53:45,721 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249625721"}]},"ts":"1676249625721"} 2023-02-13 00:53:45,725 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 00:53:45,732 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 00:53:45,736 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 569 msec 2023-02-13 00:53:45,778 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 00:53:45,780 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 00:53:45,780 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:45,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 00:53:45,833 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 00:53:45,837 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 00:53:45,842 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 33 msec 2023-02-13 00:53:45,869 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 00:53:45,885 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 00:53:45,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 42 msec 2023-02-13 00:53:45,906 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 00:53:45,910 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 00:53:45,913 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.705sec 2023-02-13 00:53:45,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 00:53:45,918 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 00:53:45,919 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 00:53:45,920 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35917,1676249621835-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 00:53:45,923 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35917,1676249621835-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 00:53:45,935 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 00:53:45,978 DEBUG [Listener at localhost/37931] zookeeper.ReadOnlyZKClient(139): Connect 0x45c0677c to 127.0.0.1:51121 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:45,983 DEBUG [Listener at localhost/37931] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1572bf4c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:45,998 DEBUG [hconnection-0x497a89ab-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 00:53:46,009 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33240, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 00:53:46,018 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:46,024 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-13 00:53:46,025 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:46,025 INFO [Listener at localhost/37931] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-13 00:53:46,026 INFO [Listener at localhost/37931] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-13 00:53:46,035 DEBUG [Listener at localhost/37931] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 00:53:46,067 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 00:53:46,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:46,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-13 00:53:46,085 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 00:53:46,087 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 00:53:46,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-13 00:53:46,090 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableOne/456f6c53b67d74b224773334332c3258 2023-02-13 00:53:46,090 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableOne/456f6c53b67d74b224773334332c3258 empty. 2023-02-13 00:53:46,093 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableOne/456f6c53b67d74b224773334332c3258 2023-02-13 00:53:46,093 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-13 00:53:46,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 00:53:46,116 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:46,118 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => 456f6c53b67d74b224773334332c3258, NAME => 'restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp 2023-02-13 00:53:46,135 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:46,135 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing 456f6c53b67d74b224773334332c3258, disabling compactions & flushes 2023-02-13 00:53:46,135 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:46,135 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:46,135 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. after waiting 0 ms 2023-02-13 00:53:46,135 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:46,135 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:46,135 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for 456f6c53b67d74b224773334332c3258: Waiting for close lock at 1676249626135Disabling compacts and flushes for region at 1676249626135Disabling writes for close at 1676249626135Writing region close event to WAL at 1676249626135Closed at 1676249626135 2023-02-13 00:53:46,139 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 00:53:46,141 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249626140"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676249626140"}]},"ts":"1676249626140"} 2023-02-13 00:53:46,143 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 00:53:46,145 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 00:53:46,145 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249626145"}]},"ts":"1676249626145"} 2023-02-13 00:53:46,147 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-13 00:53:46,153 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:46,154 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:46,154 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:46,154 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:46,154 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:46,154 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=456f6c53b67d74b224773334332c3258, ASSIGN}] 2023-02-13 00:53:46,156 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=456f6c53b67d74b224773334332c3258, ASSIGN 2023-02-13 00:53:46,157 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=456f6c53b67d74b224773334332c3258, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36081,1676249623105; forceNewPlan=false, retain=false 2023-02-13 00:53:46,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 00:53:46,308 INFO [jenkins-hbase4:35917] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 00:53:46,309 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=456f6c53b67d74b224773334332c3258, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:46,309 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249626309"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249626309"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249626309"}]},"ts":"1676249626309"} 2023-02-13 00:53:46,312 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 456f6c53b67d74b224773334332c3258, server=jenkins-hbase4.apache.org,36081,1676249623105}] 2023-02-13 00:53:46,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 00:53:46,466 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:46,466 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 00:53:46,469 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37996, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 00:53:46,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:46,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 456f6c53b67d74b224773334332c3258, NAME => 'restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:46,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne 456f6c53b67d74b224773334332c3258 2023-02-13 00:53:46,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:46,476 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 456f6c53b67d74b224773334332c3258 2023-02-13 00:53:46,476 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 456f6c53b67d74b224773334332c3258 2023-02-13 00:53:46,477 INFO [StoreOpener-456f6c53b67d74b224773334332c3258-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 456f6c53b67d74b224773334332c3258 2023-02-13 00:53:46,479 DEBUG [StoreOpener-456f6c53b67d74b224773334332c3258-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableOne/456f6c53b67d74b224773334332c3258/family 2023-02-13 00:53:46,480 DEBUG [StoreOpener-456f6c53b67d74b224773334332c3258-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableOne/456f6c53b67d74b224773334332c3258/family 2023-02-13 00:53:46,480 INFO [StoreOpener-456f6c53b67d74b224773334332c3258-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 456f6c53b67d74b224773334332c3258 columnFamilyName family 2023-02-13 00:53:46,481 INFO [StoreOpener-456f6c53b67d74b224773334332c3258-1] regionserver.HStore(310): Store=456f6c53b67d74b224773334332c3258/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:46,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableOne/456f6c53b67d74b224773334332c3258 2023-02-13 00:53:46,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableOne/456f6c53b67d74b224773334332c3258 2023-02-13 00:53:46,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 456f6c53b67d74b224773334332c3258 2023-02-13 00:53:46,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableOne/456f6c53b67d74b224773334332c3258/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:46,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 456f6c53b67d74b224773334332c3258; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64817070, jitterRate=-0.034150391817092896}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:46,492 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 456f6c53b67d74b224773334332c3258: Running coprocessor pre-open hook at 1676249626476Writing region info on filesystem at 1676249626476Initializing all the Stores at 1676249626477 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249626477Cleaning up temporary data from old regions at 1676249626484 (+7 ms)Cleaning up detritus from prior splits at 1676249626485 (+1 ms)Running coprocessor post-open hooks at 1676249626491 (+6 ms)Region opened successfully at 1676249626491 2023-02-13 00:53:46,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258., pid=11, masterSystemTime=1676249626466 2023-02-13 00:53:46,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:46,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:46,497 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=456f6c53b67d74b224773334332c3258, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:46,498 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249626497"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249626497"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249626497"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249626497"}]},"ts":"1676249626497"} 2023-02-13 00:53:46,503 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-13 00:53:46,503 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 456f6c53b67d74b224773334332c3258, server=jenkins-hbase4.apache.org,36081,1676249623105 in 188 msec 2023-02-13 00:53:46,506 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-13 00:53:46,506 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=456f6c53b67d74b224773334332c3258, ASSIGN in 349 msec 2023-02-13 00:53:46,507 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 00:53:46,508 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249626508"}]},"ts":"1676249626508"} 2023-02-13 00:53:46,510 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-13 00:53:46,514 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 00:53:46,517 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 434 msec 2023-02-13 00:53:46,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 00:53:46,709 INFO [Listener at localhost/37931] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-13 00:53:46,709 DEBUG [Listener at localhost/37931] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-13 00:53:46,711 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:46,715 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-13 00:53:46,715 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:46,715 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-13 00:53:46,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:46,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-13 00:53:46,721 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 00:53:46,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-13 00:53:46,723 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 00:53:46,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 00:53:46,725 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:46,726 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3 empty. 2023-02-13 00:53:46,727 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:46,727 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-13 00:53:46,759 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:46,761 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => 22c2a498eeb3bc3c7a7a3e9b852f9ed3, NAME => 'restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp 2023-02-13 00:53:46,785 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:46,785 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing 22c2a498eeb3bc3c7a7a3e9b852f9ed3, disabling compactions & flushes 2023-02-13 00:53:46,785 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:46,785 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:46,785 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. after waiting 0 ms 2023-02-13 00:53:46,785 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:46,785 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:46,785 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for 22c2a498eeb3bc3c7a7a3e9b852f9ed3: Waiting for close lock at 1676249626785Disabling compacts and flushes for region at 1676249626785Disabling writes for close at 1676249626785Writing region close event to WAL at 1676249626785Closed at 1676249626785 2023-02-13 00:53:46,791 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 00:53:46,793 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249626793"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676249626793"}]},"ts":"1676249626793"} 2023-02-13 00:53:46,795 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 00:53:46,797 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 00:53:46,797 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249626797"}]},"ts":"1676249626797"} 2023-02-13 00:53:46,799 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-13 00:53:46,803 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:46,804 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:46,804 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:46,804 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:46,804 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:46,805 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=22c2a498eeb3bc3c7a7a3e9b852f9ed3, ASSIGN}] 2023-02-13 00:53:46,807 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=22c2a498eeb3bc3c7a7a3e9b852f9ed3, ASSIGN 2023-02-13 00:53:46,808 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=22c2a498eeb3bc3c7a7a3e9b852f9ed3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46237,1676249623051; forceNewPlan=false, retain=false 2023-02-13 00:53:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 00:53:46,958 INFO [jenkins-hbase4:35917] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 00:53:46,960 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=22c2a498eeb3bc3c7a7a3e9b852f9ed3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:46,960 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249626959"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249626959"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249626959"}]},"ts":"1676249626959"} 2023-02-13 00:53:46,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 22c2a498eeb3bc3c7a7a3e9b852f9ed3, server=jenkins-hbase4.apache.org,46237,1676249623051}] 2023-02-13 00:53:47,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 00:53:47,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:47,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 22c2a498eeb3bc3c7a7a3e9b852f9ed3, NAME => 'restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:47,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:47,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:47,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:47,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:47,127 INFO [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:47,129 DEBUG [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3/family 2023-02-13 00:53:47,129 DEBUG [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3/family 2023-02-13 00:53:47,130 INFO [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 22c2a498eeb3bc3c7a7a3e9b852f9ed3 columnFamilyName family 2023-02-13 00:53:47,130 INFO [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] regionserver.HStore(310): Store=22c2a498eeb3bc3c7a7a3e9b852f9ed3/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:47,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:47,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:47,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:47,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:47,141 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 22c2a498eeb3bc3c7a7a3e9b852f9ed3; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68146849, jitterRate=0.01546718180179596}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:47,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 22c2a498eeb3bc3c7a7a3e9b852f9ed3: Running coprocessor pre-open hook at 1676249627125Writing region info on filesystem at 1676249627125Initializing all the Stores at 1676249627126 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249627126Cleaning up temporary data from old regions at 1676249627135 (+9 ms)Cleaning up detritus from prior splits at 1676249627135Running coprocessor post-open hooks at 1676249627141 (+6 ms)Region opened successfully at 1676249627141 2023-02-13 00:53:47,142 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3., pid=14, masterSystemTime=1676249627116 2023-02-13 00:53:47,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:47,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:47,146 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=22c2a498eeb3bc3c7a7a3e9b852f9ed3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:47,147 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249627146"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249627146"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249627146"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249627146"}]},"ts":"1676249627146"} 2023-02-13 00:53:47,153 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-13 00:53:47,153 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 22c2a498eeb3bc3c7a7a3e9b852f9ed3, server=jenkins-hbase4.apache.org,46237,1676249623051 in 187 msec 2023-02-13 00:53:47,157 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-13 00:53:47,157 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=22c2a498eeb3bc3c7a7a3e9b852f9ed3, ASSIGN in 349 msec 2023-02-13 00:53:47,159 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 00:53:47,159 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249627159"}]},"ts":"1676249627159"} 2023-02-13 00:53:47,161 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-13 00:53:47,165 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 00:53:47,168 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 448 msec 2023-02-13 00:53:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 00:53:47,331 INFO [Listener at localhost/37931] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-13 00:53:47,331 DEBUG [Listener at localhost/37931] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-13 00:53:47,331 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:47,336 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-13 00:53:47,336 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:47,336 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-13 00:53:47,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:47,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-13 00:53:47,345 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 00:53:47,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-13 00:53:47,346 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 00:53:47,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 00:53:47,349 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableThree/550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:47,350 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableThree/550d39def51a6efaca2ce06801ec5dfb empty. 2023-02-13 00:53:47,350 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableThree/550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:47,350 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-13 00:53:47,381 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:47,383 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => 550d39def51a6efaca2ce06801ec5dfb, NAME => 'restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.tmp 2023-02-13 00:53:47,403 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:47,403 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing 550d39def51a6efaca2ce06801ec5dfb, disabling compactions & flushes 2023-02-13 00:53:47,403 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:47,403 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:47,403 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. after waiting 0 ms 2023-02-13 00:53:47,403 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:47,403 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:47,404 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for 550d39def51a6efaca2ce06801ec5dfb: Waiting for close lock at 1676249627403Disabling compacts and flushes for region at 1676249627403Disabling writes for close at 1676249627403Writing region close event to WAL at 1676249627403Closed at 1676249627403 2023-02-13 00:53:47,408 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 00:53:47,410 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676249627409"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676249627409"}]},"ts":"1676249627409"} 2023-02-13 00:53:47,412 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 00:53:47,414 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 00:53:47,414 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249627414"}]},"ts":"1676249627414"} 2023-02-13 00:53:47,416 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-13 00:53:47,420 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:47,421 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:47,422 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:47,422 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:47,422 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:47,422 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=550d39def51a6efaca2ce06801ec5dfb, ASSIGN}] 2023-02-13 00:53:47,424 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=550d39def51a6efaca2ce06801ec5dfb, ASSIGN 2023-02-13 00:53:47,425 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=550d39def51a6efaca2ce06801ec5dfb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36081,1676249623105; forceNewPlan=false, retain=false 2023-02-13 00:53:47,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 00:53:47,576 INFO [jenkins-hbase4:35917] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 00:53:47,577 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=550d39def51a6efaca2ce06801ec5dfb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:47,577 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676249627577"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249627577"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249627577"}]},"ts":"1676249627577"} 2023-02-13 00:53:47,580 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 550d39def51a6efaca2ce06801ec5dfb, server=jenkins-hbase4.apache.org,36081,1676249623105}] 2023-02-13 00:53:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 00:53:47,738 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:47,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 550d39def51a6efaca2ce06801ec5dfb, NAME => 'restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:47,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:47,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:47,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:47,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:47,741 INFO [StoreOpener-550d39def51a6efaca2ce06801ec5dfb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:47,743 DEBUG [StoreOpener-550d39def51a6efaca2ce06801ec5dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableThree/550d39def51a6efaca2ce06801ec5dfb/family 2023-02-13 00:53:47,743 DEBUG [StoreOpener-550d39def51a6efaca2ce06801ec5dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableThree/550d39def51a6efaca2ce06801ec5dfb/family 2023-02-13 00:53:47,744 INFO [StoreOpener-550d39def51a6efaca2ce06801ec5dfb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 550d39def51a6efaca2ce06801ec5dfb columnFamilyName family 2023-02-13 00:53:47,745 INFO [StoreOpener-550d39def51a6efaca2ce06801ec5dfb-1] regionserver.HStore(310): Store=550d39def51a6efaca2ce06801ec5dfb/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:47,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableThree/550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:47,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableThree/550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:47,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:47,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableThree/550d39def51a6efaca2ce06801ec5dfb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:47,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 550d39def51a6efaca2ce06801ec5dfb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59137707, jitterRate=-0.1187794953584671}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:47,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 550d39def51a6efaca2ce06801ec5dfb: Running coprocessor pre-open hook at 1676249627739Writing region info on filesystem at 1676249627739Initializing all the Stores at 1676249627741 (+2 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249627741Cleaning up temporary data from old regions at 1676249627749 (+8 ms)Cleaning up detritus from prior splits at 1676249627749Running coprocessor post-open hooks at 1676249627759 (+10 ms)Region opened successfully at 1676249627759 2023-02-13 00:53:47,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb., pid=17, masterSystemTime=1676249627733 2023-02-13 00:53:47,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:47,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:47,764 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=550d39def51a6efaca2ce06801ec5dfb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:47,764 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676249627764"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249627764"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249627764"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249627764"}]},"ts":"1676249627764"} 2023-02-13 00:53:47,769 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-13 00:53:47,769 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 550d39def51a6efaca2ce06801ec5dfb, server=jenkins-hbase4.apache.org,36081,1676249623105 in 187 msec 2023-02-13 00:53:47,771 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-13 00:53:47,772 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=550d39def51a6efaca2ce06801ec5dfb, ASSIGN in 347 msec 2023-02-13 00:53:47,773 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 00:53:47,773 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249627773"}]},"ts":"1676249627773"} 2023-02-13 00:53:47,775 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-13 00:53:47,779 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 00:53:47,781 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 440 msec 2023-02-13 00:53:47,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35917] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 00:53:47,952 INFO [Listener at localhost/37931] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-13 00:53:47,952 DEBUG [Listener at localhost/37931] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-13 00:53:47,953 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:47,957 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-13 00:53:47,957 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:47,957 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-13 00:53:47,958 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:47,963 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:47,967 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:47,971 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:47,972 INFO [Listener at localhost/37931] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 00:53:47,981 INFO [Listener at localhost/37931] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 00:53:47,981 INFO [Listener at localhost/37931] master.TestRetainAssignmentOnRestart(157): Stopping HMaster and jenkins-hbase4.apache.org,46237,1676249623051 server 2023-02-13 00:53:47,981 INFO [Listener at localhost/37931] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase4:35917,5,FailOnTimeoutGroup] 2023-02-13 00:53:47,981 INFO [Listener at localhost/37931] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35917,1676249621835' ***** 2023-02-13 00:53:47,981 INFO [Listener at localhost/37931] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-13 00:53:47,982 DEBUG [M:0;jenkins-hbase4:35917] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fed4055, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:47,983 INFO [M:0;jenkins-hbase4:35917] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:47,984 DEBUG [M:0;jenkins-hbase4:35917] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fa48e27 to 127.0.0.1:51121 2023-02-13 00:53:47,986 DEBUG [M:0;jenkins-hbase4:35917] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:47,987 INFO [M:0;jenkins-hbase4:35917] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35917,1676249621835; all regions closed. 2023-02-13 00:53:47,987 DEBUG [M:0;jenkins-hbase4:35917] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:47,988 DEBUG [M:0;jenkins-hbase4:35917] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 00:53:47,988 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 00:53:47,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249624311] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249624311,5,FailOnTimeoutGroup] 2023-02-13 00:53:47,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249624309] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249624309,5,FailOnTimeoutGroup] 2023-02-13 00:53:47,988 DEBUG [M:0;jenkins-hbase4:35917] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 00:53:47,988 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:47,990 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:47,990 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:47,990 INFO [M:0;jenkins-hbase4:35917] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 00:53:47,990 INFO [M:0;jenkins-hbase4:35917] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 00:53:47,990 INFO [M:0;jenkins-hbase4:35917] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 00:53:47,991 DEBUG [M:0;jenkins-hbase4:35917] master.HMaster(1502): Stopping service threads 2023-02-13 00:53:47,990 INFO [Listener at localhost/37931] hbase.LocalHBaseCluster(360): Waiting on jenkins-hbase4.apache.org,35917,1676249621835 2023-02-13 00:53:47,989 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:47,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:47,992 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:47,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:47,991 INFO [M:0;jenkins-hbase4:35917] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 00:53:47,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:47,993 INFO [M:0;jenkins-hbase4:35917] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 00:53:47,993 ERROR [M:0;jenkins-hbase4:35917] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] 2023-02-13 00:53:47,994 INFO [M:0;jenkins-hbase4:35917] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 00:53:47,994 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 00:53:48,093 DEBUG [M:0;jenkins-hbase4:35917] zookeeper.ZKUtil(398): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 00:53:48,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:48,093 WARN [M:0;jenkins-hbase4:35917] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 00:53:48,094 INFO [M:0;jenkins-hbase4:35917] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 00:53:48,095 INFO [M:0;jenkins-hbase4:35917] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 00:53:48,095 DEBUG [M:0;jenkins-hbase4:35917] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 00:53:48,095 INFO [M:0;jenkins-hbase4:35917] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:48,096 DEBUG [M:0;jenkins-hbase4:35917] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:48,096 DEBUG [M:0;jenkins-hbase4:35917] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 00:53:48,096 DEBUG [M:0;jenkins-hbase4:35917] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:48,096 INFO [M:0;jenkins-hbase4:35917] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.19 KB heapSize=76.14 KB 2023-02-13 00:53:48,164 INFO [M:0;jenkins-hbase4:35917] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.19 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2175da5d91e346d19124c03d59fc9c10 2023-02-13 00:53:48,204 DEBUG [M:0;jenkins-hbase4:35917] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2175da5d91e346d19124c03d59fc9c10 as hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2175da5d91e346d19124c03d59fc9c10 2023-02-13 00:53:48,213 INFO [M:0;jenkins-hbase4:35917] regionserver.HStore(1080): Added hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2175da5d91e346d19124c03d59fc9c10, entries=17, sequenceid=141, filesize=8.5 K 2023-02-13 00:53:48,215 INFO [M:0;jenkins-hbase4:35917] regionserver.HRegion(2947): Finished flush of dataSize ~64.19 KB/65731, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 119ms, sequenceid=141, compaction requested=false 2023-02-13 00:53:48,218 INFO [M:0;jenkins-hbase4:35917] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:48,219 DEBUG [M:0;jenkins-hbase4:35917] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676249628095Disabling compacts and flushes for region at 1676249628095Disabling writes for close at 1676249628096 (+1 ms)Obtaining lock to block concurrent updates at 1676249628096Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676249628096Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65731, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676249628106 (+10 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676249628106Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676249628109 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676249628136 (+27 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676249628139 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676249628206 (+67 ms)Finished flush of dataSize ~64.19 KB/65731, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 119ms, sequenceid=141, compaction requested=false at 1676249628215 (+9 ms)Writing region close event to WAL at 1676249628218 (+3 ms)Closed at 1676249628218 2023-02-13 00:53:48,226 INFO [M:0;jenkins-hbase4:35917] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 00:53:48,226 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:48,226 INFO [M:0;jenkins-hbase4:35917] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35917 2023-02-13 00:53:48,235 WARN [RS-EventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35917 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:53:48,236 DEBUG [M:0;jenkins-hbase4:35917] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,35917,1676249621835 already deleted, retry=false 2023-02-13 00:53:48,236 DEBUG [RS-EventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35917 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 2023-02-13 00:53:48,237 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 00:53:48,308 WARN [RS-EventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35917 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:53:48,309 DEBUG [RS-EventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35917 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 2023-02-13 00:53:48,310 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 00:53:48,311 WARN [RS-EventLoopGroup-5-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35917 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:53:48,311 DEBUG [RS-EventLoopGroup-5-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35917 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 2023-02-13 00:53:48,311 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 00:53:48,341 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:48,341 INFO [M:0;jenkins-hbase4:35917] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35917,1676249621835; zookeeper connection closed. 2023-02-13 00:53:48,342 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2df5741b rejected from java.util.concurrent.ThreadPoolExecutor@7d4e93fa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:48,342 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:35917-0x1004b2b4b940000, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:48,342 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@694131ab rejected from java.util.concurrent.ThreadPoolExecutor@7d4e93fa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:48,343 INFO [Listener at localhost/37931] hbase.MiniHBaseCluster(508): Stopping Thread[RS:0;jenkins-hbase4:46237,5,FailOnTimeoutGroup] 2023-02-13 00:53:48,343 INFO [Listener at localhost/37931] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46237,1676249623051' ***** 2023-02-13 00:53:48,343 INFO [Listener at localhost/37931] regionserver.HRegionServer(2310): STOPPED: Stopping rs 0 2023-02-13 00:53:48,343 INFO [Listener at localhost/37931] master.TestRetainAssignmentOnRestart(163): Sleeping a bit 2023-02-13 00:53:48,408 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:53:48,438 INFO [RS:0;jenkins-hbase4:46237] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:53:48,438 INFO [RS:0;jenkins-hbase4:46237] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:53:48,438 INFO [RS:0;jenkins-hbase4:46237] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:53:48,438 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(3304): Received CLOSE for 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:48,439 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(3304): Received CLOSE for c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:48,440 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:48,440 DEBUG [RS:0;jenkins-hbase4:46237] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fc6fdf1 to 127.0.0.1:51121 2023-02-13 00:53:48,440 DEBUG [RS:0;jenkins-hbase4:46237] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:48,440 INFO [RS:0;jenkins-hbase4:46237] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:53:48,441 INFO [RS:0;jenkins-hbase4:46237] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:53:48,441 INFO [RS:0;jenkins-hbase4:46237] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:53:48,441 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 00:53:48,441 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1475): Waiting on 3 regions to close 2023-02-13 00:53:48,442 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1479): Online Regions={22c2a498eeb3bc3c7a7a3e9b852f9ed3=restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3., c7ca26b901efc47292da4c15f8e352a8=hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 00:53:48,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 22c2a498eeb3bc3c7a7a3e9b852f9ed3, disabling compactions & flushes 2023-02-13 00:53:48,443 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 00:53:48,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:48,443 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 00:53:48,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:48,443 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 00:53:48,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. after waiting 0 ms 2023-02-13 00:53:48,443 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 00:53:48,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:48,444 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 00:53:48,444 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=5.14 KB heapSize=9.42 KB 2023-02-13 00:53:48,445 DEBUG [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1505): Waiting on 1588230740, 22c2a498eeb3bc3c7a7a3e9b852f9ed3, c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:48,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 00:53:48,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:48,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 22c2a498eeb3bc3c7a7a3e9b852f9ed3: Waiting for close lock at 1676249628441Running coprocessor pre-close hooks at 1676249628441Disabling compacts and flushes for region at 1676249628441Disabling writes for close at 1676249628444 (+3 ms)Writing region close event to WAL at 1676249628444Running coprocessor post-close hooks at 1676249628454 (+10 ms)Closed at 1676249628454 2023-02-13 00:53:48,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:48,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c7ca26b901efc47292da4c15f8e352a8, disabling compactions & flushes 2023-02-13 00:53:48,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:48,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:48,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. after waiting 0 ms 2023-02-13 00:53:48,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:48,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c7ca26b901efc47292da4c15f8e352a8 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 00:53:48,487 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.77 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/.tmp/info/124d767dbca74c5e8944113a79bb87be 2023-02-13 00:53:48,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/.tmp/info/b0875349412a4ebf99a16037b4f46f12 2023-02-13 00:53:48,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/.tmp/info/b0875349412a4ebf99a16037b4f46f12 as hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/info/b0875349412a4ebf99a16037b4f46f12 2023-02-13 00:53:48,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/info/b0875349412a4ebf99a16037b4f46f12, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 00:53:48,511 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:48,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c7ca26b901efc47292da4c15f8e352a8 in 57ms, sequenceid=6, compaction requested=false 2023-02-13 00:53:48,514 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/.tmp/table/c61e0d9c17a2475eb1e9a77840a1809f 2023-02-13 00:53:48,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 00:53:48,524 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/.tmp/info/124d767dbca74c5e8944113a79bb87be as hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info/124d767dbca74c5e8944113a79bb87be 2023-02-13 00:53:48,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 00:53:48,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:48,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c7ca26b901efc47292da4c15f8e352a8: Waiting for close lock at 1676249628455Running coprocessor pre-close hooks at 1676249628455Disabling compacts and flushes for region at 1676249628455Disabling writes for close at 1676249628456 (+1 ms)Obtaining lock to block concurrent updates at 1676249628456Preparing flush snapshotting stores in c7ca26b901efc47292da4c15f8e352a8 at 1676249628456Finished memstore snapshotting hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676249628456Flushing stores of hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. at 1676249628458 (+2 ms)Flushing c7ca26b901efc47292da4c15f8e352a8/info: creating writer at 1676249628458Flushing c7ca26b901efc47292da4c15f8e352a8/info: appending metadata at 1676249628462 (+4 ms)Flushing c7ca26b901efc47292da4c15f8e352a8/info: closing flushed file at 1676249628462Flushing c7ca26b901efc47292da4c15f8e352a8/info: reopening flushed file at 1676249628501 (+39 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c7ca26b901efc47292da4c15f8e352a8 in 57ms, sequenceid=6, compaction requested=false at 1676249628513 (+12 ms)Writing region close event to WAL at 1676249628519 (+6 ms)Running coprocessor post-close hooks at 1676249628525 (+6 ms)Closed at 1676249628525 2023-02-13 00:53:48,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:48,531 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info/124d767dbca74c5e8944113a79bb87be, entries=40, sequenceid=24, filesize=9.5 K 2023-02-13 00:53:48,534 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/.tmp/table/c61e0d9c17a2475eb1e9a77840a1809f as hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/table/c61e0d9c17a2475eb1e9a77840a1809f 2023-02-13 00:53:48,542 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/table/c61e0d9c17a2475eb1e9a77840a1809f, entries=8, sequenceid=24, filesize=5.0 K 2023-02-13 00:53:48,544 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 100ms, sequenceid=24, compaction requested=false 2023-02-13 00:53:48,544 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 00:53:48,544 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 00:53:48,544 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 00:53:48,554 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=1 2023-02-13 00:53:48,554 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 00:53:48,555 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 00:53:48,555 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676249628443Running coprocessor pre-close hooks at 1676249628443Disabling compacts and flushes for region at 1676249628443Disabling writes for close at 1676249628444 (+1 ms)Obtaining lock to block concurrent updates at 1676249628444Preparing flush snapshotting stores in 1588230740 at 1676249628444Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=5262, getHeapSize=9600, getOffHeapSize=0, getCellsCount=48 at 1676249628445 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676249628448 (+3 ms)Flushing 1588230740/info: creating writer at 1676249628448Flushing 1588230740/info: appending metadata at 1676249628456 (+8 ms)Flushing 1588230740/info: closing flushed file at 1676249628456Flushing 1588230740/table: creating writer at 1676249628497 (+41 ms)Flushing 1588230740/table: appending metadata at 1676249628501 (+4 ms)Flushing 1588230740/table: closing flushed file at 1676249628501Flushing 1588230740/info: reopening flushed file at 1676249628525 (+24 ms)Flushing 1588230740/table: reopening flushed file at 1676249628535 (+10 ms)Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 100ms, sequenceid=24, compaction requested=false at 1676249628544 (+9 ms)Writing region close event to WAL at 1676249628549 (+5 ms)Running coprocessor post-close hooks at 1676249628554 (+5 ms)Closed at 1676249628555 (+1 ms) 2023-02-13 00:53:48,555 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 00:53:48,645 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46237,1676249623051; all regions closed. 2023-02-13 00:53:48,656 DEBUG [RS:0;jenkins-hbase4:46237] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs 2023-02-13 00:53:48,656 INFO [RS:0;jenkins-hbase4:46237] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46237%2C1676249623051.meta:.meta(num 1676249624861) 2023-02-13 00:53:48,664 DEBUG [RS:0;jenkins-hbase4:46237] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs 2023-02-13 00:53:48,664 INFO [RS:0;jenkins-hbase4:46237] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46237%2C1676249623051:(num 1676249624672) 2023-02-13 00:53:48,664 DEBUG [RS:0;jenkins-hbase4:46237] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:48,664 INFO [RS:0;jenkins-hbase4:46237] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:48,664 INFO [RS:0;jenkins-hbase4:46237] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 00:53:48,665 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:48,665 INFO [RS:0;jenkins-hbase4:46237] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46237 2023-02-13 00:53:48,669 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:48,669 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:48,669 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:48,669 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:48,669 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:48,669 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@a61d3e6 rejected from java.util.concurrent.ThreadPoolExecutor@e5e4672[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:48,670 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:48,670 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7ea73389 rejected from java.util.concurrent.ThreadPoolExecutor@e5e4672[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:48,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:48,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:48,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:48,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:48,671 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,46237,1676249623051 znode expired, triggering replicatorRemoved event 2023-02-13 00:53:48,671 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,46237,1676249623051 znode expired, triggering replicatorRemoved event 2023-02-13 00:53:48,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:48,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:48,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:48,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:48,771 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:48,771 INFO [RS:0;jenkins-hbase4:46237] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46237,1676249623051; zookeeper connection closed. 2023-02-13 00:53:48,771 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6cc85a7 rejected from java.util.concurrent.ThreadPoolExecutor@e5e4672[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:48,771 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b940001, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:48,772 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5af3cd20] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5af3cd20 2023-02-13 00:53:48,772 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6c8f630e rejected from java.util.concurrent.ThreadPoolExecutor@e5e4672[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:49,313 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 00:53:49,314 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 00:53:50,316 DEBUG [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 00:53:50,317 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 00:53:50,343 INFO [Listener at localhost/37931] master.TestRetainAssignmentOnRestart(166): Starting HMaster and region server jenkins-hbase4.apache.org,46237,1676249623051 second time with the same port 2023-02-13 00:53:50,360 INFO [Listener at localhost/37931] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:53:50,360 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:50,360 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:50,360 INFO [Listener at localhost/37931] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:50,360 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:50,360 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:50,360 INFO [Listener at localhost/37931] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:50,364 INFO [Listener at localhost/37931] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38569 2023-02-13 00:53:50,365 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:50,366 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:50,367 INFO [Listener at localhost/37931] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38569 connecting to ZooKeeper ensemble=127.0.0.1:51121 2023-02-13 00:53:50,371 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:385690x0, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:50,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:38569-0x1004b2b4b940009 connected 2023-02-13 00:53:50,376 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:50,376 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(162): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 00:53:50,377 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:50,377 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38569 2023-02-13 00:53:50,381 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38569 2023-02-13 00:53:50,382 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38569 2023-02-13 00:53:50,382 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38569 2023-02-13 00:53:50,382 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38569 2023-02-13 00:53:50,382 INFO [Listener at localhost/37931] master.HMaster(439): hbase.rootdir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc, hbase.cluster.distributed=false 2023-02-13 00:53:50,384 DEBUG [pool-218-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-13 00:53:50,388 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:50,389 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 00:53:50,390 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:50,391 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:50,391 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:50,391 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:50,392 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:50,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:53:50,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:53:50,393 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,38569,1676249630359 from backup master directory 2023-02-13 00:53:50,394 INFO [M:0;jenkins-hbase4:38569] regionserver.HRegionServer(952): ClusterId : a53384a2-dda6-4252-aa83-796fb497e498 2023-02-13 00:53:50,395 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:50,395 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:50,395 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 00:53:50,395 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:50,399 INFO [Listener at localhost/37931] client.ConnectionUtils(127): regionserver/jenkins-hbase4:46237 server-side Connection retries=45 2023-02-13 00:53:50,399 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:50,399 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:50,399 INFO [Listener at localhost/37931] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:50,400 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:50,400 INFO [Listener at localhost/37931] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:50,401 INFO [Listener at localhost/37931] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:50,402 INFO [Listener at localhost/37931] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46237 2023-02-13 00:53:50,403 INFO [Listener at localhost/37931] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:53:50,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:50,425 DEBUG [Listener at localhost/37931] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:53:50,426 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:50,428 INFO [Listener at localhost/37931] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:50,430 INFO [Listener at localhost/37931] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46237 connecting to ZooKeeper ensemble=127.0.0.1:51121 2023-02-13 00:53:50,448 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:462370x0, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:50,449 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(162): regionserver:462370x0, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:53:50,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46237-0x1004b2b4b94000a connected 2023-02-13 00:53:50,451 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(162): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 00:53:50,452 DEBUG [Listener at localhost/37931] zookeeper.ZKUtil(164): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:50,452 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46237 2023-02-13 00:53:50,453 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46237 2023-02-13 00:53:50,453 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46237 2023-02-13 00:53:50,453 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46237 2023-02-13 00:53:50,454 DEBUG [Listener at localhost/37931] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46237 2023-02-13 00:53:50,455 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(952): ClusterId : a53384a2-dda6-4252-aa83-796fb497e498 2023-02-13 00:53:50,457 DEBUG [RS:3;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:53:50,461 DEBUG [RS:3;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:53:50,461 DEBUG [RS:3;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:53:50,464 DEBUG [RS:3;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:53:50,465 DEBUG [RS:3;jenkins-hbase4:46237] zookeeper.ReadOnlyZKClient(139): Connect 0x37e0fc94 to 127.0.0.1:51121 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:50,465 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 00:53:50,490 DEBUG [RS:3;jenkins-hbase4:46237] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39271418, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:50,490 DEBUG [RS:3;jenkins-hbase4:46237] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d1897e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:50,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x587bb366 to 127.0.0.1:51121 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:50,502 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d4edd6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:50,502 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:50,503 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 00:53:50,505 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase4:46237 2023-02-13 00:53:50,505 INFO [RS:3;jenkins-hbase4:46237] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:53:50,505 INFO [RS:3;jenkins-hbase4:46237] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:53:50,505 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:53:50,506 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38569,1676249630359 with isa=jenkins-hbase4.apache.org/172.31.14.131:46237, startcode=1676249630398 2023-02-13 00:53:50,506 DEBUG [RS:3;jenkins-hbase4:46237] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:53:50,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:50,512 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56811, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:53:50,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:53:50,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,35917,1676249621835 to hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,35917,1676249621835-dead as it is dead 2023-02-13 00:53:50,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,35917,1676249621835-dead/jenkins-hbase4.apache.org%2C35917%2C1676249621835.1676249623645 2023-02-13 00:53:50,528 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,35917,1676249621835-dead/jenkins-hbase4.apache.org%2C35917%2C1676249621835.1676249623645 after 4ms 2023-02-13 00:53:50,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,35917,1676249621835-dead/jenkins-hbase4.apache.org%2C35917%2C1676249621835.1676249623645 to hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C35917%2C1676249621835.1676249623645 2023-02-13 00:53:50,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,35917,1676249621835-dead 2023-02-13 00:53:50,529 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:50,532 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38569%2C1676249630359, suffix=, logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,38569,1676249630359, archiveDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/oldWALs, maxLogs=10 2023-02-13 00:53:50,537 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 00:53:50,538 WARN [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 00:53:50,561 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK] 2023-02-13 00:53:50,562 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK] 2023-02-13 00:53:50,563 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK] 2023-02-13 00:53:50,563 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'restartTableThree' 2023-02-13 00:53:50,564 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'restartTableOne' 2023-02-13 00:53:50,568 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/WALs/jenkins-hbase4.apache.org,38569,1676249630359/jenkins-hbase4.apache.org%2C38569%2C1676249630359.1676249630533 2023-02-13 00:53:50,568 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK], DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK], DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK]] 2023-02-13 00:53:50,568 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:50,569 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:50,569 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:50,569 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:50,571 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:50,572 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 00:53:50,573 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 00:53:50,581 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2175da5d91e346d19124c03d59fc9c10 2023-02-13 00:53:50,581 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:50,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-13 00:53:50,583 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C35917%2C1676249621835.1676249623645 2023-02-13 00:53:50,598 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C35917%2C1676249621835.1676249623645 2023-02-13 00:53:50,599 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C35917%2C1676249621835.1676249623645 2023-02-13 00:53:50,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:50,606 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-13 00:53:50,607 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61278171, jitterRate=-0.08688409626483917}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:50,607 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676249630569Initializing all the Stores at 1676249630570 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249630570Cleaning up temporary data from old regions at 1676249630600 (+30 ms)Cleaning up detritus from prior splits at 1676249630601 (+1 ms)Region opened successfully at 1676249630607 (+6 ms) 2023-02-13 00:53:50,607 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 00:53:50,608 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 00:53:50,608 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 00:53:50,609 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 00:53:50,609 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 00:53:50,618 DEBUG [RS:2;jenkins-hbase4:38169] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:53:50,620 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34985, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:53:50,620 DEBUG [RS:1;jenkins-hbase4:36081] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:53:50,621 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38569] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:53:50,622 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59907, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:53:50,622 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38569] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:53:50,623 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-13 00:53:50,624 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-13 00:53:50,624 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-13 00:53:50,624 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-13 00:53:50,624 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-13 00:53:50,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-13 00:53:50,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-13 00:53:50,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 16 msec 2023-02-13 00:53:50,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 00:53:50,626 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-13 00:53:50,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,46237,1676249623051, table=hbase:meta, region=1588230740 2023-02-13 00:53:50,628 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-13 00:53:50,629 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:50,629 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:50,631 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46237,1676249623051 already deleted, retry=false 2023-02-13 00:53:50,631 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,46237,1676249623051 on jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:50,638 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,46237,1676249623051, splitWal=true, meta=true 2023-02-13 00:53:50,638 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase4.apache.org,46237,1676249623051 (carryingMeta=true) jenkins-hbase4.apache.org,46237,1676249623051/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2c61594f[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 00:53:50,638 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38569,1676249630359 with isa=jenkins-hbase4.apache.org/172.31.14.131:46237, startcode=1676249630398 2023-02-13 00:53:50,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:53:50,639 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 00:53:50,639 WARN [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 200 ms and then retrying. 2023-02-13 00:53:50,648 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 00:53:50,649 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 00:53:50,649 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-13 00:53:50,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 00:53:50,650 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 00:53:50,651 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 00:53:50,651 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 00:53:50,652 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 00:53:50,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,38569,1676249630359, sessionid=0x1004b2b4b940009, setting cluster-up flag (Was=true) 2023-02-13 00:53:50,655 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 00:53:50,656 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:50,658 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 00:53:50,659 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:50,660 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/.hbase-snapshot/.tmp 2023-02-13 00:53:50,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:50,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:50,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:50,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:50,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 00:53:50,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:50,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676249660662 2023-02-13 00:53:50,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 00:53:50,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 00:53:50,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 00:53:50,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 00:53:50,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 00:53:50,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 00:53:50,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 00:53:50,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 00:53:50,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 00:53:50,665 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 00:53:50,665 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 00:53:50,667 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249630665,5,FailOnTimeoutGroup] 2023-02-13 00:53:50,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249630667,5,FailOnTimeoutGroup] 2023-02-13 00:53:50,668 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,668 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 00:53:50,668 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46237,1676249623051; numProcessing=1 2023-02-13 00:53:50,668 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,668 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,669 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676249630390, completionTime=-1 2023-02-13 00:53:50,669 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 00:53:50,669 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46237,1676249623051, splitWal=true, meta=true 2023-02-13 00:53:50,670 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46237,1676249623051, splitWal=true, meta=true, isMeta: true 2023-02-13 00:53:50,672 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051-splitting 2023-02-13 00:53:50,673 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051-splitting dir is empty, no logs to split. 2023-02-13 00:53:50,673 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,46237,1676249623051 WAL count=0, meta=true 2023-02-13 00:53:50,676 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051-splitting dir is empty, no logs to split. 2023-02-13 00:53:50,676 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,46237,1676249623051 WAL count=0, meta=true 2023-02-13 00:53:50,676 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,46237,1676249623051 WAL splitting is done? wals=0, meta=true 2023-02-13 00:53:50,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 00:53:50,679 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 00:53:50,680 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,46237,1676249623051; forceNewPlan=false, retain=false 2023-02-13 00:53:50,831 DEBUG [jenkins-hbase4:38569] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:50,835 DEBUG [jenkins-hbase4:38569] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:50,835 DEBUG [jenkins-hbase4:38569] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:50,835 DEBUG [jenkins-hbase4:38569] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:50,835 DEBUG [jenkins-hbase4:38569] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:50,837 INFO [jenkins-hbase4:38569] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 0 retained the pre-restart assignment. 1 regions were assigned to random hosts, since the old hosts for these regions are no longer present in the cluster. These hosts were: 2023-02-13 00:53:50,837 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36081,1676249623105, state=OPENING 2023-02-13 00:53:50,839 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 00:53:50,839 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 00:53:50,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36081,1676249623105}] 2023-02-13 00:53:50,842 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38569,1676249630359 with isa=jenkins-hbase4.apache.org/172.31.14.131:46237, startcode=1676249630398 2023-02-13 00:53:50,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:50,844 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc 2023-02-13 00:53:50,844 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45541 2023-02-13 00:53:50,844 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:53:50,846 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:50,846 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:50,846 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:50,847 DEBUG [RS:3;jenkins-hbase4:46237] zookeeper.ZKUtil(162): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:50,847 WARN [RS:3;jenkins-hbase4:46237] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:50,847 INFO [RS:3;jenkins-hbase4:46237] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:50,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:50,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:50,847 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:50,848 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46237,1676249630398] 2023-02-13 00:53:50,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:50,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:50,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:50,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:50,852 DEBUG [RS:3;jenkins-hbase4:46237] zookeeper.ZKUtil(162): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:50,852 DEBUG [RS:3;jenkins-hbase4:46237] zookeeper.ZKUtil(162): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:50,852 DEBUG [RS:3;jenkins-hbase4:46237] zookeeper.ZKUtil(162): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:50,853 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:53:50,854 INFO [RS:3;jenkins-hbase4:46237] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:53:50,855 INFO [RS:3;jenkins-hbase4:46237] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:53:50,856 INFO [RS:3;jenkins-hbase4:46237] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:53:50,856 INFO [RS:3;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,856 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:53:50,857 INFO [RS:3;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,857 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:46237, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,858 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:46237, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,858 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,858 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,858 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:46237, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,858 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:46237, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:50,858 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:46237, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,858 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:46237, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,858 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:46237, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,858 DEBUG [RS:3;jenkins-hbase4:46237] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:46237, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:50,859 INFO [RS:3;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,859 INFO [RS:3;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,859 INFO [RS:3;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,869 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=200ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 00:53:50,869 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPENING, ts=1676249630839, server=jenkins-hbase4.apache.org,36081,1676249623105}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 00:53:50,870 INFO [RS:3;jenkins-hbase4:46237] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:53:50,870 INFO [RS:3;jenkins-hbase4:46237] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46237,1676249630398-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:50,880 INFO [RS:3;jenkins-hbase4:46237] regionserver.Replication(203): jenkins-hbase4.apache.org,46237,1676249630398 started 2023-02-13 00:53:50,881 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46237,1676249630398, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46237, sessionid=0x1004b2b4b94000a 2023-02-13 00:53:50,881 DEBUG [RS:3;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:53:50,881 DEBUG [RS:3;jenkins-hbase4:46237] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:50,881 DEBUG [RS:3;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46237,1676249630398' 2023-02-13 00:53:50,881 DEBUG [RS:3;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:53:50,881 DEBUG [RS:3;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:53:50,881 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:50,881 DEBUG [RS:3;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:53:50,881 DEBUG [RS:3;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:53:50,881 DEBUG [RS:3;jenkins-hbase4:46237] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:50,881 DEBUG [RS:3;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46237,1676249630398' 2023-02-13 00:53:50,882 DEBUG [RS:3;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:53:50,882 DEBUG [RS:3;jenkins-hbase4:46237] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:53:50,882 DEBUG [RS:3;jenkins-hbase4:46237] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:53:50,882 INFO [RS:3;jenkins-hbase4:46237] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:53:50,882 INFO [RS:3;jenkins-hbase4:46237] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:53:50,883 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33252, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 00:53:50,884 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46237] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:33252 deadline: 1676249690884, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:50,985 INFO [RS:3;jenkins-hbase4:46237] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46237%2C1676249630398, suffix=, logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249630398, archiveDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs, maxLogs=32 2023-02-13 00:53:50,987 WARN [ReadOnlyZKClient-127.0.0.1:51121@0x45c0677c] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-02-13 00:53:50,988 DEBUG [Listener at localhost/37931] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 00:53:50,992 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38008, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 00:53:50,993 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36081] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:38008 deadline: 1676249690992, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:50,996 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:50,996 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 00:53:50,998 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38016, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=AdminService 2023-02-13 00:53:51,004 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 00:53:51,004 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:51,013 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK] 2023-02-13 00:53:51,013 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK] 2023-02-13 00:53:51,013 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK] 2023-02-13 00:53:51,015 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36081%2C1676249623105.meta, suffix=.meta, logDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,36081,1676249623105, archiveDir=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs, maxLogs=32 2023-02-13 00:53:51,020 INFO [RS:3;jenkins-hbase4:46237] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249630398/jenkins-hbase4.apache.org%2C46237%2C1676249630398.1676249630986 2023-02-13 00:53:51,021 DEBUG [RS:3;jenkins-hbase4:46237] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK], DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK], DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK]] 2023-02-13 00:53:51,038 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK] 2023-02-13 00:53:51,038 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK] 2023-02-13 00:53:51,038 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK] 2023-02-13 00:53:51,044 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,36081,1676249623105/jenkins-hbase4.apache.org%2C36081%2C1676249623105.meta.1676249631016.meta 2023-02-13 00:53:51,044 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43395,DS-012f2b2e-6845-4514-b01e-82821451c519,DISK], DatanodeInfoWithStorage[127.0.0.1:39413,DS-8d798703-a000-4606-9d1c-5bb60d2f414a,DISK], DatanodeInfoWithStorage[127.0.0.1:41349,DS-a5773ebc-c5ee-46a5-8ef4-c86cb854ff6e,DISK]] 2023-02-13 00:53:51,045 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:51,045 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 00:53:51,045 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 00:53:51,046 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 00:53:51,046 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 00:53:51,046 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:51,046 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 00:53:51,047 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 00:53:51,048 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 00:53:51,049 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info 2023-02-13 00:53:51,049 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info 2023-02-13 00:53:51,050 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 00:53:51,058 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info/124d767dbca74c5e8944113a79bb87be 2023-02-13 00:53:51,058 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:51,058 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 00:53:51,059 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:53:51,059 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:53:51,060 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 00:53:51,061 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:51,061 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 00:53:51,062 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/table 2023-02-13 00:53:51,062 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/table 2023-02-13 00:53:51,063 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 00:53:51,071 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/table/c61e0d9c17a2475eb1e9a77840a1809f 2023-02-13 00:53:51,071 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:51,072 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740 2023-02-13 00:53:51,074 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740 2023-02-13 00:53:51,077 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 00:53:51,078 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 00:53:51,079 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=28; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71358975, jitterRate=0.06333158910274506}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 00:53:51,079 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676249631047Writing region info on filesystem at 1676249631047Initializing all the Stores at 1676249631048 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249631048Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249631048Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249631048Cleaning up temporary data from old regions at 1676249631075 (+27 ms)Cleaning up detritus from prior splits at 1676249631076 (+1 ms)Running coprocessor post-open hooks at 1676249631079 (+3 ms)Region opened successfully at 1676249631079 2023-02-13 00:53:51,080 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=20, masterSystemTime=1676249630996 2023-02-13 00:53:51,084 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 00:53:51,084 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 00:53:51,085 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36081,1676249623105, state=OPEN 2023-02-13 00:53:51,088 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 00:53:51,089 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 00:53:51,091 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-02-13 00:53:51,091 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36081,1676249623105 in 250 msec 2023-02-13 00:53:51,094 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-02-13 00:53:51,094 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 414 msec 2023-02-13 00:53:51,208 WARN [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35917 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:53:51,209 WARN [Listener at localhost/37931] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:35917 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:53:51,209 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35917 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35917 2023-02-13 00:53:51,210 DEBUG [Listener at localhost/37931] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35917 this server is in the failed servers list 2023-02-13 00:53:51,210 WARN [Listener at localhost/37931] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:35917 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:35917 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnSingleRSRestart(TestRetainAssignmentOnRestart.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:35917 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-13 00:53:51,211 DEBUG [Listener at localhost/37931] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 00:53:51,213 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35772, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 00:53:51,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:53:51,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.CallRunner(144): callId: 50 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:35772 deadline: 1676249691217, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:53:51,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:53:51,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.CallRunner(144): callId: 52 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:35772 deadline: 1676249691319, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:53:51,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:53:51,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.CallRunner(144): callId: 54 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:35772 deadline: 1676249691521, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:53:51,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:53:51,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.CallRunner(144): callId: 56 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:35772 deadline: 1676249691824, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:53:51,870 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 00:53:51,873 DEBUG [hconnection-0x46b28b1f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 00:53:51,875 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38028, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=ClientService 2023-02-13 00:53:51,878 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=c7ca26b901efc47292da4c15f8e352a8, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,46237,1676249623051, regionLocation=jenkins-hbase4.apache.org,46237,1676249623051, openSeqNum=2 2023-02-13 00:53:51,878 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=456f6c53b67d74b224773334332c3258, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36081,1676249623105, regionLocation=jenkins-hbase4.apache.org,36081,1676249623105, openSeqNum=2 2023-02-13 00:53:51,878 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=550d39def51a6efaca2ce06801ec5dfb, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36081,1676249623105, regionLocation=jenkins-hbase4.apache.org,36081,1676249623105, openSeqNum=2 2023-02-13 00:53:51,879 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=22c2a498eeb3bc3c7a7a3e9b852f9ed3, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,46237,1676249623051, regionLocation=jenkins-hbase4.apache.org,46237,1676249623051, openSeqNum=2 2023-02-13 00:53:51,879 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 00:53:51,879 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676249691879 2023-02-13 00:53:51,879 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676249751879 2023-02-13 00:53:51,879 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 9 msec 2023-02-13 00:53:51,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38569,1676249630359-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:51,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38569,1676249630359-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:51,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38569,1676249630359-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:51,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:38569, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:51,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:51,903 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,46237,1676249623051 had 3 regions 2023-02-13 00:53:51,903 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. is NOT online; state={c7ca26b901efc47292da4c15f8e352a8 state=OPEN, ts=1676249631878, server=jenkins-hbase4.apache.org,46237,1676249623051}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 00:53:51,906 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46237,1676249623051, splitWal=true, meta=true, isMeta: false 2023-02-13 00:53:51,909 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051-splitting dir is empty, no logs to split. 2023-02-13 00:53:51,909 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,46237,1676249623051 WAL count=0, meta=false 2023-02-13 00:53:51,912 WARN [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase4.apache.org,46237,1676249623051/hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8., unknown_server=jenkins-hbase4.apache.org,46237,1676249623051/restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:51,912 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,46237,1676249623051-splitting dir is empty, no logs to split. 2023-02-13 00:53:51,912 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,46237,1676249623051 WAL count=0, meta=false 2023-02-13 00:53:51,912 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,46237,1676249623051 WAL splitting is done? wals=0, meta=false 2023-02-13 00:53:51,915 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c7ca26b901efc47292da4c15f8e352a8, ASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=22c2a498eeb3bc3c7a7a3e9b852f9ed3, ASSIGN}] 2023-02-13 00:53:51,916 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c7ca26b901efc47292da4c15f8e352a8, ASSIGN 2023-02-13 00:53:51,916 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=22c2a498eeb3bc3c7a7a3e9b852f9ed3, ASSIGN 2023-02-13 00:53:51,917 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:51,918 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46237,1676249623051 2023-02-13 00:53:51,918 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c7ca26b901efc47292da4c15f8e352a8, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,46237,1676249623051; forceNewPlan=false, retain=true 2023-02-13 00:53:51,918 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=22c2a498eeb3bc3c7a7a3e9b852f9ed3, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,46237,1676249623051; forceNewPlan=false, retain=true 2023-02-13 00:53:51,995 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 00:53:51,995 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-13 00:53:52,068 INFO [jenkins-hbase4:38569] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-13 00:53:52,069 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=22c2a498eeb3bc3c7a7a3e9b852f9ed3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:52,069 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c7ca26b901efc47292da4c15f8e352a8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:52,070 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249632069"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249632069"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249632069"}]},"ts":"1676249632069"} 2023-02-13 00:53:52,070 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249632069"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249632069"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249632069"}]},"ts":"1676249632069"} 2023-02-13 00:53:52,073 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; OpenRegionProcedure 22c2a498eeb3bc3c7a7a3e9b852f9ed3, server=jenkins-hbase4.apache.org,46237,1676249630398}] 2023-02-13 00:53:52,074 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; OpenRegionProcedure c7ca26b901efc47292da4c15f8e352a8, server=jenkins-hbase4.apache.org,46237,1676249630398}] 2023-02-13 00:53:52,225 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:52,226 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 00:53:52,227 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33262, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=AdminService 2023-02-13 00:53:52,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:52,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(7850): Opening region: {ENCODED => 22c2a498eeb3bc3c7a7a3e9b852f9ed3, NAME => 'restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:52,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:52,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:52,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(7890): checking encryption for 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:52,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(7893): checking classloading for 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:52,234 INFO [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:52,235 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:52,235 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(7850): Opening region: {ENCODED => c7ca26b901efc47292da4c15f8e352a8, NAME => 'hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:52,236 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:52,236 DEBUG [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3/family 2023-02-13 00:53:52,236 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:52,236 DEBUG [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3/family 2023-02-13 00:53:52,236 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(7890): checking encryption for c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:52,236 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(7893): checking classloading for c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:52,238 INFO [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 22c2a498eeb3bc3c7a7a3e9b852f9ed3 columnFamilyName family 2023-02-13 00:53:52,238 INFO [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:52,239 INFO [StoreOpener-22c2a498eeb3bc3c7a7a3e9b852f9ed3-1] regionserver.HStore(310): Store=22c2a498eeb3bc3c7a7a3e9b852f9ed3/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:52,240 DEBUG [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/info 2023-02-13 00:53:52,240 DEBUG [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/info 2023-02-13 00:53:52,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:52,240 INFO [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7ca26b901efc47292da4c15f8e352a8 columnFamilyName info 2023-02-13 00:53:52,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:52,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1054): writing seq id for 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:52,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1071): Opened 22c2a498eeb3bc3c7a7a3e9b852f9ed3; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62489877, jitterRate=-0.0688282698392868}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:52,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(964): Region open journal for 22c2a498eeb3bc3c7a7a3e9b852f9ed3: Running coprocessor pre-open hook at 1676249632233Writing region info on filesystem at 1676249632233Initializing all the Stores at 1676249632234 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249632234Cleaning up temporary data from old regions at 1676249632242 (+8 ms)Cleaning up detritus from prior splits at 1676249632243 (+1 ms)Running coprocessor post-open hooks at 1676249632246 (+3 ms)Region opened successfully at 1676249632246 2023-02-13 00:53:52,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3., pid=23, masterSystemTime=1676249632225 2023-02-13 00:53:52,251 DEBUG [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] regionserver.HStore(539): loaded hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/info/b0875349412a4ebf99a16037b4f46f12 2023-02-13 00:53:52,252 INFO [StoreOpener-c7ca26b901efc47292da4c15f8e352a8-1] regionserver.HStore(310): Store=c7ca26b901efc47292da4c15f8e352a8/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:52,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:52,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:46237-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:52,253 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:52,253 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=22c2a498eeb3bc3c7a7a3e9b852f9ed3, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:52,254 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249632253"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249632253"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249632253"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249632253"}]},"ts":"1676249632253"} 2023-02-13 00:53:52,255 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:52,259 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-02-13 00:53:52,260 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1054): writing seq id for c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:52,260 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; OpenRegionProcedure 22c2a498eeb3bc3c7a7a3e9b852f9ed3, server=jenkins-hbase4.apache.org,46237,1676249630398 in 183 msec 2023-02-13 00:53:52,261 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1071): Opened c7ca26b901efc47292da4c15f8e352a8; next sequenceid=10; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70835665, jitterRate=0.05553366243839264}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:52,261 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(964): Region open journal for c7ca26b901efc47292da4c15f8e352a8: Running coprocessor pre-open hook at 1676249632236Writing region info on filesystem at 1676249632236Initializing all the Stores at 1676249632237 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249632237Cleaning up temporary data from old regions at 1676249632255 (+18 ms)Cleaning up detritus from prior splits at 1676249632256 (+1 ms)Running coprocessor post-open hooks at 1676249632261 (+5 ms)Region opened successfully at 1676249632261 2023-02-13 00:53:52,261 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=22c2a498eeb3bc3c7a7a3e9b852f9ed3, ASSIGN in 345 msec 2023-02-13 00:53:52,262 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8., pid=24, masterSystemTime=1676249632225 2023-02-13 00:53:52,264 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:52,264 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46237-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:52,265 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c7ca26b901efc47292da4c15f8e352a8, regionState=OPEN, openSeqNum=10, regionLocation=jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:52,265 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249632265"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249632265"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249632265"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249632265"}]},"ts":"1676249632265"} 2023-02-13 00:53:52,269 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-02-13 00:53:52,269 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; OpenRegionProcedure c7ca26b901efc47292da4c15f8e352a8, server=jenkins-hbase4.apache.org,46237,1676249630398 in 193 msec 2023-02-13 00:53:52,272 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=18 2023-02-13 00:53:52,273 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,46237,1676249623051 after splitting done 2023-02-13 00:53:52,273 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c7ca26b901efc47292da4c15f8e352a8, ASSIGN in 354 msec 2023-02-13 00:53:52,273 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,46237,1676249623051 from processing; numProcessing=0 2023-02-13 00:53:52,276 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,46237,1676249623051, splitWal=true, meta=true in 1.6410 sec 2023-02-13 00:53:52,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:53:52,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38569] ipc.CallRunner(144): callId: 58 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:35772 deadline: 1676249692331, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:53:52,903 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-13 00:53:52,908 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 00:53:52,910 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33268, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=ClientService 2023-02-13 00:53:52,924 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 00:53:52,928 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 00:53:52,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.533sec 2023-02-13 00:53:52,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 00:53:52,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 00:53:52,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 00:53:52,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38569,1676249630359-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 00:53:52,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38569,1676249630359-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 00:53:52,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 00:53:53,347 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:53,361 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:53,368 INFO [Listener at localhost/37931] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:53,368 INFO [Listener at localhost/37931] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 00:53:53,372 INFO [Listener at localhost/37931] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 00:53:53,372 INFO [Listener at localhost/37931] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 456f6c53b67d74b224773334332c3258, NAME => 'restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,36081,1676249623105, currentServer=jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:53,372 INFO [Listener at localhost/37931] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 550d39def51a6efaca2ce06801ec5dfb, NAME => 'restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,36081,1676249623105, currentServer=jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:53,372 INFO [Listener at localhost/37931] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => c7ca26b901efc47292da4c15f8e352a8, NAME => 'hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,46237,1676249623051, currentServer=jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:53,372 INFO [Listener at localhost/37931] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 22c2a498eeb3bc3c7a7a3e9b852f9ed3, NAME => 'restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,46237,1676249623051, currentServer=jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:53,372 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 00:53:53,372 INFO [Listener at localhost/37931] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 00:53:53,373 DEBUG [Listener at localhost/37931] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45c0677c to 127.0.0.1:51121 2023-02-13 00:53:53,373 DEBUG [Listener at localhost/37931] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:53,373 DEBUG [Listener at localhost/37931] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 00:53:53,373 DEBUG [Listener at localhost/37931] util.JVMClusterUtil(257): Found active master hash=934750728, stopped=false 2023-02-13 00:53:53,373 INFO [Listener at localhost/37931] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:53,376 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:53,376 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:53,376 INFO [Listener at localhost/37931] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 00:53:53,376 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:53,376 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:53,376 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:53,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:53,378 DEBUG [Listener at localhost/37931] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x587bb366 to 127.0.0.1:51121 2023-02-13 00:53:53,379 DEBUG [Listener at localhost/37931] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:53,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:53,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:53,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:53,379 INFO [Listener at localhost/37931] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36081,1676249623105' ***** 2023-02-13 00:53:53,379 INFO [Listener at localhost/37931] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 00:53:53,379 INFO [Listener at localhost/37931] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38169,1676249623143' ***** 2023-02-13 00:53:53,380 INFO [Listener at localhost/37931] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 00:53:53,380 INFO [RS:1;jenkins-hbase4:36081] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:53:53,380 INFO [Listener at localhost/37931] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46237,1676249630398' ***** 2023-02-13 00:53:53,380 INFO [RS:2;jenkins-hbase4:38169] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:53:53,380 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:53:53,380 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:53:53,380 INFO [Listener at localhost/37931] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 00:53:53,380 INFO [RS:1;jenkins-hbase4:36081] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:53:53,381 INFO [RS:2;jenkins-hbase4:38169] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:53:53,382 INFO [RS:1;jenkins-hbase4:36081] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:53:53,382 INFO [RS:3;jenkins-hbase4:46237] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:53:53,382 INFO [RS:2;jenkins-hbase4:38169] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:53:53,382 INFO [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:53,382 DEBUG [RS:2;jenkins-hbase4:38169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x563f9214 to 127.0.0.1:51121 2023-02-13 00:53:53,382 INFO [RS:3;jenkins-hbase4:46237] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:53:53,382 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:53:53,382 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(3304): Received CLOSE for 550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:53,382 INFO [RS:3;jenkins-hbase4:46237] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:53:53,383 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(3304): Received CLOSE for 22c2a498eeb3bc3c7a7a3e9b852f9ed3 2023-02-13 00:53:53,383 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(3304): Received CLOSE for 456f6c53b67d74b224773334332c3258 2023-02-13 00:53:53,383 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:53,382 DEBUG [RS:2;jenkins-hbase4:38169] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:53,383 DEBUG [RS:1;jenkins-hbase4:36081] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4eabd27d to 127.0.0.1:51121 2023-02-13 00:53:53,383 INFO [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38169,1676249623143; all regions closed. 2023-02-13 00:53:53,383 DEBUG [RS:1;jenkins-hbase4:36081] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:53,383 INFO [RS:1;jenkins-hbase4:36081] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:53:53,383 INFO [RS:1;jenkins-hbase4:36081] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:53:53,383 INFO [RS:1;jenkins-hbase4:36081] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:53:53,383 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 00:53:53,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 550d39def51a6efaca2ce06801ec5dfb, disabling compactions & flushes 2023-02-13 00:53:53,389 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(3304): Received CLOSE for c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:53,389 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:53,389 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1475): Waiting on 3 regions to close 2023-02-13 00:53:53,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:53,389 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:53,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. after waiting 0 ms 2023-02-13 00:53:53,389 DEBUG [RS:3;jenkins-hbase4:46237] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37e0fc94 to 127.0.0.1:51121 2023-02-13 00:53:53,389 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1479): Online Regions={550d39def51a6efaca2ce06801ec5dfb=restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb., 1588230740=hbase:meta,,1.1588230740, 456f6c53b67d74b224773334332c3258=restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258.} 2023-02-13 00:53:53,390 DEBUG [RS:3;jenkins-hbase4:46237] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:53,390 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 00:53:53,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:53,391 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 00:53:53,391 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1603): Closing 22c2a498eeb3bc3c7a7a3e9b852f9ed3, disabling compactions & flushes 2023-02-13 00:53:53,390 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 00:53:53,391 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1479): Online Regions={22c2a498eeb3bc3c7a7a3e9b852f9ed3=restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3., c7ca26b901efc47292da4c15f8e352a8=hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8.} 2023-02-13 00:53:53,391 DEBUG [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1505): Waiting on 22c2a498eeb3bc3c7a7a3e9b852f9ed3, c7ca26b901efc47292da4c15f8e352a8 2023-02-13 00:53:53,390 DEBUG [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1505): Waiting on 1588230740, 456f6c53b67d74b224773334332c3258, 550d39def51a6efaca2ce06801ec5dfb 2023-02-13 00:53:53,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:53,391 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 00:53:53,391 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:53,391 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 00:53:53,391 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. after waiting 0 ms 2023-02-13 00:53:53,391 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 00:53:53,391 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:53,392 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=1.89 KB heapSize=3.81 KB 2023-02-13 00:53:53,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableThree/550d39def51a6efaca2ce06801ec5dfb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 00:53:53,404 DEBUG [RS:2;jenkins-hbase4:38169] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs 2023-02-13 00:53:53,404 INFO [RS:2;jenkins-hbase4:38169] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38169%2C1676249623143:(num 1676249624672) 2023-02-13 00:53:53,404 DEBUG [RS:2;jenkins-hbase4:38169] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:53,404 INFO [RS:2;jenkins-hbase4:38169] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:53,404 INFO [RS:2;jenkins-hbase4:38169] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 00:53:53,404 INFO [RS:2;jenkins-hbase4:38169] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:53:53,404 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:53,404 INFO [RS:2;jenkins-hbase4:38169] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:53:53,405 INFO [RS:2;jenkins-hbase4:38169] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:53:53,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableTwo/22c2a498eeb3bc3c7a7a3e9b852f9ed3/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 00:53:53,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:53,405 INFO [RS:2;jenkins-hbase4:38169] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38169 2023-02-13 00:53:53,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 550d39def51a6efaca2ce06801ec5dfb: Waiting for close lock at 1676249633383Running coprocessor pre-close hooks at 1676249633383Disabling compacts and flushes for region at 1676249633383Disabling writes for close at 1676249633389 (+6 ms)Writing region close event to WAL at 1676249633391 (+2 ms)Running coprocessor post-close hooks at 1676249633404 (+13 ms)Closed at 1676249633405 (+1 ms) 2023-02-13 00:53:53,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676249627338.550d39def51a6efaca2ce06801ec5dfb. 2023-02-13 00:53:53,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 456f6c53b67d74b224773334332c3258, disabling compactions & flushes 2023-02-13 00:53:53,410 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:53,410 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:53,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:53,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1557): Region close journal for 22c2a498eeb3bc3c7a7a3e9b852f9ed3: Waiting for close lock at 1676249633390Running coprocessor pre-close hooks at 1676249633390Disabling compacts and flushes for region at 1676249633390Disabling writes for close at 1676249633391 (+1 ms)Writing region close event to WAL at 1676249633392 (+1 ms)Running coprocessor post-close hooks at 1676249633410 (+18 ms)Closed at 1676249633410 2023-02-13 00:53:53,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. after waiting 0 ms 2023-02-13 00:53:53,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:53,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676249626717.22c2a498eeb3bc3c7a7a3e9b852f9ed3. 2023-02-13 00:53:53,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1603): Closing c7ca26b901efc47292da4c15f8e352a8, disabling compactions & flushes 2023-02-13 00:53:53,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:53,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:53,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. after waiting 0 ms 2023-02-13 00:53:53,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:53,411 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:53,411 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:53,411 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38169,1676249623143 2023-02-13 00:53:53,411 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:53,411 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4587e489 rejected from java.util.concurrent.ThreadPoolExecutor@41b868e1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:53,411 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:53,411 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:53,411 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:53,412 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f2d393f rejected from java.util.concurrent.ThreadPoolExecutor@41b868e1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:53,415 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38169,1676249623143] 2023-02-13 00:53:53,415 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38169,1676249623143; numProcessing=1 2023-02-13 00:53:53,416 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:53,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/default/restartTableOne/456f6c53b67d74b224773334332c3258/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 00:53:53,418 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38169,1676249623143 already deleted, retry=false 2023-02-13 00:53:53,418 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38169,1676249623143 expired; onlineServers=2 2023-02-13 00:53:53,419 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:53,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:53,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 456f6c53b67d74b224773334332c3258: Waiting for close lock at 1676249633409Running coprocessor pre-close hooks at 1676249633409Disabling compacts and flushes for region at 1676249633409Disabling writes for close at 1676249633410 (+1 ms)Writing region close event to WAL at 1676249633411 (+1 ms)Running coprocessor post-close hooks at 1676249633418 (+7 ms)Closed at 1676249633419 (+1 ms) 2023-02-13 00:53:53,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676249626076.456f6c53b67d74b224773334332c3258. 2023-02-13 00:53:53,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/namespace/c7ca26b901efc47292da4c15f8e352a8/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=9 2023-02-13 00:53:53,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:53,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] regionserver.HRegion(1557): Region close journal for c7ca26b901efc47292da4c15f8e352a8: Waiting for close lock at 1676249633410Running coprocessor pre-close hooks at 1676249633410Disabling compacts and flushes for region at 1676249633410Disabling writes for close at 1676249633411 (+1 ms)Writing region close event to WAL at 1676249633415 (+4 ms)Running coprocessor post-close hooks at 1676249633423 (+8 ms)Closed at 1676249633424 (+1 ms) 2023-02-13 00:53:53,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46237-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676249625161.c7ca26b901efc47292da4c15f8e352a8. 2023-02-13 00:53:53,428 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.89 KB at sequenceid=34 (bloomFilter=false), to=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/.tmp/info/c0d548770b8c4229a3fd1e7c1e236f81 2023-02-13 00:53:53,436 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/.tmp/info/c0d548770b8c4229a3fd1e7c1e236f81 as hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info/c0d548770b8c4229a3fd1e7c1e236f81 2023-02-13 00:53:53,443 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/info/c0d548770b8c4229a3fd1e7c1e236f81, entries=16, sequenceid=34, filesize=6.6 K 2023-02-13 00:53:53,445 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.89 KB/1932, heapSize ~3.30 KB/3376, currentSize=0 B/0 for 1588230740 in 52ms, sequenceid=34, compaction requested=false 2023-02-13 00:53:53,452 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/data/hbase/meta/1588230740/recovered.edits/37.seqid, newMaxSeqId=37, maxSeqId=27 2023-02-13 00:53:53,453 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 00:53:53,453 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 00:53:53,453 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676249633389Running coprocessor pre-close hooks at 1676249633389Disabling compacts and flushes for region at 1676249633389Disabling writes for close at 1676249633391 (+2 ms)Obtaining lock to block concurrent updates at 1676249633392 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676249633392Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=1932, getHeapSize=3856, getOffHeapSize=0, getCellsCount=16 at 1676249633394 (+2 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676249633397 (+3 ms)Flushing 1588230740/info: creating writer at 1676249633397Flushing 1588230740/info: appending metadata at 1676249633404 (+7 ms)Flushing 1588230740/info: closing flushed file at 1676249633404Flushing 1588230740/info: reopening flushed file at 1676249633437 (+33 ms)Finished flush of dataSize ~1.89 KB/1932, heapSize ~3.30 KB/3376, currentSize=0 B/0 for 1588230740 in 52ms, sequenceid=34, compaction requested=false at 1676249633445 (+8 ms)Writing region close event to WAL at 1676249633448 (+3 ms)Running coprocessor post-close hooks at 1676249633453 (+5 ms)Closed at 1676249633453 2023-02-13 00:53:53,453 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 00:53:53,461 INFO [regionserver/jenkins-hbase4:46237.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:53,530 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 00:53:53,530 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 00:53:53,576 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:53,576 INFO [RS:2;jenkins-hbase4:38169] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38169,1676249623143; zookeeper connection closed. 2023-02-13 00:53:53,576 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3789a6c0 rejected from java.util.concurrent.ThreadPoolExecutor@41b868e1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:53,577 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:38169-0x1004b2b4b940003, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:53,577 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4138d625] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4138d625 2023-02-13 00:53:53,577 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@42d72891 rejected from java.util.concurrent.ThreadPoolExecutor@41b868e1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:53,591 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46237,1676249630398; all regions closed. 2023-02-13 00:53:53,591 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36081,1676249623105; all regions closed. 2023-02-13 00:53:53,597 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/WALs/jenkins-hbase4.apache.org,36081,1676249623105/jenkins-hbase4.apache.org%2C36081%2C1676249623105.meta.1676249631016.meta not finished, retry = 0 2023-02-13 00:53:53,599 DEBUG [RS:3;jenkins-hbase4:46237] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs 2023-02-13 00:53:53,600 INFO [RS:3;jenkins-hbase4:46237] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46237%2C1676249630398:(num 1676249630986) 2023-02-13 00:53:53,600 DEBUG [RS:3;jenkins-hbase4:46237] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:53,600 INFO [RS:3;jenkins-hbase4:46237] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:53,600 INFO [RS:3;jenkins-hbase4:46237] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:46237 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 00:53:53,600 INFO [RS:3;jenkins-hbase4:46237] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:53:53,600 INFO [regionserver/jenkins-hbase4:46237.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:53,600 INFO [RS:3;jenkins-hbase4:46237] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:53:53,600 INFO [RS:3;jenkins-hbase4:46237] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:53:53,601 INFO [RS:3;jenkins-hbase4:46237] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46237 2023-02-13 00:53:53,603 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:53,603 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46237,1676249630398 2023-02-13 00:53:53,603 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:53,603 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@521e6fac rejected from java.util.concurrent.ThreadPoolExecutor@5621f137[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:53,605 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46237,1676249630398] 2023-02-13 00:53:53,605 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46237,1676249630398; numProcessing=2 2023-02-13 00:53:53,606 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46237,1676249630398 already deleted, retry=false 2023-02-13 00:53:53,606 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,46237,1676249630398 expired; onlineServers=1 2023-02-13 00:53:53,700 DEBUG [RS:1;jenkins-hbase4:36081] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs 2023-02-13 00:53:53,700 INFO [RS:1;jenkins-hbase4:36081] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36081%2C1676249623105.meta:.meta(num 1676249631016) 2023-02-13 00:53:53,707 DEBUG [RS:1;jenkins-hbase4:36081] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/oldWALs 2023-02-13 00:53:53,707 INFO [RS:1;jenkins-hbase4:36081] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36081%2C1676249623105:(num 1676249624672) 2023-02-13 00:53:53,707 DEBUG [RS:1;jenkins-hbase4:36081] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:53,707 INFO [RS:1;jenkins-hbase4:36081] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:53,707 INFO [RS:1;jenkins-hbase4:36081] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 00:53:53,707 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:53,708 INFO [RS:1;jenkins-hbase4:36081] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36081 2023-02-13 00:53:53,712 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36081,1676249623105 2023-02-13 00:53:53,712 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:53,712 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@92b87dd rejected from java.util.concurrent.ThreadPoolExecutor@d141220[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:53,713 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36081,1676249623105] 2023-02-13 00:53:53,713 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36081,1676249623105; numProcessing=3 2023-02-13 00:53:53,714 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36081,1676249623105 already deleted, retry=false 2023-02-13 00:53:53,714 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36081,1676249623105 expired; onlineServers=0 2023-02-13 00:53:53,714 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38569,1676249630359' ***** 2023-02-13 00:53:53,714 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 00:53:53,715 DEBUG [M:0;jenkins-hbase4:38569] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3388b826, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:53,715 INFO [M:0;jenkins-hbase4:38569] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38569,1676249630359 2023-02-13 00:53:53,715 INFO [M:0;jenkins-hbase4:38569] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38569,1676249630359; all regions closed. 2023-02-13 00:53:53,715 DEBUG [M:0;jenkins-hbase4:38569] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:53,715 DEBUG [M:0;jenkins-hbase4:38569] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 00:53:53,715 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 00:53:53,715 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249630667] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249630667,5,FailOnTimeoutGroup] 2023-02-13 00:53:53,715 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249630665] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249630665,5,FailOnTimeoutGroup] 2023-02-13 00:53:53,715 DEBUG [M:0;jenkins-hbase4:38569] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 00:53:53,717 INFO [M:0;jenkins-hbase4:38569] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 00:53:53,717 INFO [M:0;jenkins-hbase4:38569] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 00:53:53,717 INFO [M:0;jenkins-hbase4:38569] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 00:53:53,717 DEBUG [M:0;jenkins-hbase4:38569] master.HMaster(1502): Stopping service threads 2023-02-13 00:53:53,717 INFO [M:0;jenkins-hbase4:38569] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 00:53:53,718 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:53,718 ERROR [M:0;jenkins-hbase4:38569] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-13 00:53:53,718 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:53,718 INFO [M:0;jenkins-hbase4:38569] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 00:53:53,718 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 00:53:53,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:53,718 DEBUG [M:0;jenkins-hbase4:38569] zookeeper.ZKUtil(398): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 00:53:53,718 WARN [M:0;jenkins-hbase4:38569] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 00:53:53,718 INFO [M:0;jenkins-hbase4:38569] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 00:53:53,719 INFO [M:0;jenkins-hbase4:38569] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 00:53:53,719 DEBUG [M:0;jenkins-hbase4:38569] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 00:53:53,719 INFO [M:0;jenkins-hbase4:38569] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:53,719 DEBUG [M:0;jenkins-hbase4:38569] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:53,719 DEBUG [M:0;jenkins-hbase4:38569] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 00:53:53,719 DEBUG [M:0;jenkins-hbase4:38569] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:53,719 INFO [M:0;jenkins-hbase4:38569] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=20.99 KB heapSize=26.11 KB 2023-02-13 00:53:53,735 INFO [M:0;jenkins-hbase4:38569] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.99 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b175511b794e4dcaaf5ce4a118760f9b 2023-02-13 00:53:53,742 DEBUG [M:0;jenkins-hbase4:38569] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b175511b794e4dcaaf5ce4a118760f9b as hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b175511b794e4dcaaf5ce4a118760f9b 2023-02-13 00:53:53,748 INFO [M:0;jenkins-hbase4:38569] regionserver.HStore(1080): Added hdfs://localhost:45541/user/jenkins/test-data/e2b9ccd1-0d08-ea3f-f7bb-3bb0a43723dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b175511b794e4dcaaf5ce4a118760f9b, entries=7, sequenceid=199, filesize=5.4 K 2023-02-13 00:53:53,749 INFO [M:0;jenkins-hbase4:38569] regionserver.HRegion(2947): Finished flush of dataSize ~20.99 KB/21489, heapSize ~26.09 KB/26720, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=199, compaction requested=false 2023-02-13 00:53:53,751 INFO [M:0;jenkins-hbase4:38569] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:53,751 DEBUG [M:0;jenkins-hbase4:38569] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676249633719Disabling compacts and flushes for region at 1676249633719Disabling writes for close at 1676249633719Obtaining lock to block concurrent updates at 1676249633719Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676249633719Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=21489, getHeapSize=26720, getOffHeapSize=0, getCellsCount=66 at 1676249633720 (+1 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676249633721 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676249633721Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676249633725 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676249633725Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676249633743 (+18 ms)Finished flush of dataSize ~20.99 KB/21489, heapSize ~26.09 KB/26720, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=199, compaction requested=false at 1676249633749 (+6 ms)Writing region close event to WAL at 1676249633751 (+2 ms)Closed at 1676249633751 2023-02-13 00:53:53,755 INFO [M:0;jenkins-hbase4:38569] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 00:53:53,755 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:53,755 INFO [M:0;jenkins-hbase4:38569] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38569 2023-02-13 00:53:53,759 DEBUG [M:0;jenkins-hbase4:38569] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,38569,1676249630359 already deleted, retry=false 2023-02-13 00:53:54,078 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:54,078 INFO [M:0;jenkins-hbase4:38569] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38569,1676249630359; zookeeper connection closed. 2023-02-13 00:53:54,078 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7a1d8a45 rejected from java.util.concurrent.ThreadPoolExecutor@7e2201de[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 18] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:54,079 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): master:38569-0x1004b2b4b940009, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:54,079 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c8812ae rejected from java.util.concurrent.ThreadPoolExecutor@7e2201de[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 18] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:54,178 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:54,178 INFO [RS:1;jenkins-hbase4:36081] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36081,1676249623105; zookeeper connection closed. 2023-02-13 00:53:54,178 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2c544b4d rejected from java.util.concurrent.ThreadPoolExecutor@d141220[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:54,179 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@71a6e340] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@71a6e340 2023-02-13 00:53:54,179 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:36081-0x1004b2b4b940002, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:54,179 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@543cbc75 rejected from java.util.concurrent.ThreadPoolExecutor@d141220[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:54,278 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:54,278 INFO [RS:3;jenkins-hbase4:46237] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46237,1676249630398; zookeeper connection closed. 2023-02-13 00:53:54,279 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b8de16 rejected from java.util.concurrent.ThreadPoolExecutor@5621f137[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:54,279 DEBUG [Listener at localhost/37931-EventThread] zookeeper.ZKWatcher(600): regionserver:46237-0x1004b2b4b94000a, quorum=127.0.0.1:51121, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:54,279 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3ebba78c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3ebba78c 2023-02-13 00:53:54,279 ERROR [Listener at localhost/37931-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7b8b57e3 rejected from java.util.concurrent.ThreadPoolExecutor@5621f137[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:54,280 INFO [Listener at localhost/37931] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 4 regionserver(s) complete 2023-02-13 00:53:54,280 WARN [Listener at localhost/37931] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 00:53:54,285 INFO [Listener at localhost/37931] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 00:53:54,389 WARN [BP-1045515978-172.31.14.131-1676249617163 heartbeating to localhost/127.0.0.1:45541] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 00:53:54,389 WARN [BP-1045515978-172.31.14.131-1676249617163 heartbeating to localhost/127.0.0.1:45541] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1045515978-172.31.14.131-1676249617163 (Datanode Uuid 569dd8da-56b0-4966-9330-63ec496574b2) service to localhost/127.0.0.1:45541 2023-02-13 00:53:54,391 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50/dfs/data/data5/current/BP-1045515978-172.31.14.131-1676249617163] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:53:54,391 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50/dfs/data/data6/current/BP-1045515978-172.31.14.131-1676249617163] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:53:54,395 WARN [Listener at localhost/37931] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 00:53:54,398 INFO [Listener at localhost/37931] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 00:53:54,403 WARN [BP-1045515978-172.31.14.131-1676249617163 heartbeating to localhost/127.0.0.1:45541] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 00:53:54,403 WARN [BP-1045515978-172.31.14.131-1676249617163 heartbeating to localhost/127.0.0.1:45541] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1045515978-172.31.14.131-1676249617163 (Datanode Uuid f126e0b0-7796-4bf2-8e9f-858da0a2bce8) service to localhost/127.0.0.1:45541 2023-02-13 00:53:54,404 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50/dfs/data/data3/current/BP-1045515978-172.31.14.131-1676249617163] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:53:54,404 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50/dfs/data/data4/current/BP-1045515978-172.31.14.131-1676249617163] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:53:54,405 WARN [Listener at localhost/37931] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 00:53:54,407 INFO [Listener at localhost/37931] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 00:53:54,511 WARN [BP-1045515978-172.31.14.131-1676249617163 heartbeating to localhost/127.0.0.1:45541] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 00:53:54,511 WARN [BP-1045515978-172.31.14.131-1676249617163 heartbeating to localhost/127.0.0.1:45541] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1045515978-172.31.14.131-1676249617163 (Datanode Uuid 60127b70-1772-4fff-80b6-3be9184b83aa) service to localhost/127.0.0.1:45541 2023-02-13 00:53:54,511 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50/dfs/data/data1/current/BP-1045515978-172.31.14.131-1676249617163] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:53:54,512 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/cluster_7c04bb8e-ffb6-eafd-86f9-03c6ba866f50/dfs/data/data2/current/BP-1045515978-172.31.14.131-1676249617163] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:53:54,550 INFO [Listener at localhost/37931] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 00:53:54,669 INFO [Listener at localhost/37931] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 00:53:54,719 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-13 00:53:54,734 INFO [Listener at localhost/37931] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=100 (was 7) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:45541 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:45541 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:45541 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:45541 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:45541 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:45541 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:46237.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/37931 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:45541 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@70c8758a java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:45541 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: ForkJoinPool-2-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:45541 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:45541 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:300) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:45541 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:45541 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:45541 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=507 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 326), ProcessCount=168 (was 168), AvailableMemoryMB=4090 (was 5319) 2023-02-13 00:53:54,744 INFO [Listener at localhost/37931] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=100, OpenFileDescriptor=507, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=168, AvailableMemoryMB=4090 2023-02-13 00:53:54,755 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/hadoop.log.dir so I do NOT create it in target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01 2023-02-13 00:53:54,755 WARN [Listener at localhost/37931] hbase.HBaseTestingUtility(449): hadoop.log.dir property value differs in configuration and system: Configuration=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/../logs while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/hadoop.log.dir Erasing configuration value by system value. 2023-02-13 00:53:54,755 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/hadoop.tmp.dir so I do NOT create it in target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01 2023-02-13 00:53:54,755 WARN [Listener at localhost/37931] hbase.HBaseTestingUtility(449): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-jenkins while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab514f70-9982-0ef3-b038-6e9045ab8218/hadoop.tmp.dir Erasing configuration value by system value. 2023-02-13 00:53:54,756 DEBUG [Listener at localhost/37931] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01 2023-02-13 00:53:54,756 INFO [Listener at localhost/37931] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-13 00:53:54,756 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 00:53:54,756 INFO [Listener at localhost/37931] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456, deleteOnExit=true 2023-02-13 00:53:54,756 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 00:53:54,756 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/test.cache.data in system properties and HBase conf 2023-02-13 00:53:54,756 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 00:53:54,756 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/hadoop.log.dir in system properties and HBase conf 2023-02-13 00:53:54,756 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 00:53:54,757 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 00:53:54,757 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 00:53:54,757 DEBUG [Listener at localhost/37931] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 00:53:54,757 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 00:53:54,757 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 00:53:54,757 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 00:53:54,757 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 00:53:54,757 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 00:53:54,758 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 00:53:54,758 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 00:53:54,758 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 00:53:54,758 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 00:53:54,758 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/nfs.dump.dir in system properties and HBase conf 2023-02-13 00:53:54,758 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/java.io.tmpdir in system properties and HBase conf 2023-02-13 00:53:54,758 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 00:53:54,758 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 00:53:54,758 INFO [Listener at localhost/37931] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 00:53:54,763 WARN [Listener at localhost/37931] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 00:53:54,763 WARN [Listener at localhost/37931] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 00:53:54,802 WARN [Listener at localhost/37931] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 00:53:54,804 INFO [Listener at localhost/37931] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 00:53:54,808 INFO [Listener at localhost/37931] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/java.io.tmpdir/Jetty_localhost_38645_hdfs____9u7fj0/webapp 2023-02-13 00:53:54,901 INFO [Listener at localhost/37931] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38645 2023-02-13 00:53:54,907 WARN [Listener at localhost/37931] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 00:53:54,907 WARN [Listener at localhost/37931] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 00:53:54,952 WARN [Listener at localhost/44257] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 00:53:54,970 WARN [Listener at localhost/44257] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 00:53:54,973 WARN [Listener at localhost/44257] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 00:53:54,975 INFO [Listener at localhost/44257] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 00:53:54,982 INFO [Listener at localhost/44257] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/java.io.tmpdir/Jetty_localhost_45761_datanode____jecvi0/webapp 2023-02-13 00:53:55,076 INFO [Listener at localhost/44257] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45761 2023-02-13 00:53:55,085 WARN [Listener at localhost/43017] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 00:53:55,101 WARN [Listener at localhost/43017] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 00:53:55,104 WARN [Listener at localhost/43017] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 00:53:55,105 INFO [Listener at localhost/43017] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 00:53:55,109 INFO [Listener at localhost/43017] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/java.io.tmpdir/Jetty_localhost_36457_datanode____z345bu/webapp 2023-02-13 00:53:55,201 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa41a4621609836c9: Processing first storage report for DS-17f61d75-c836-4da5-b671-d9a9297ad34b from datanode 8cf3b173-72eb-4183-a548-1c31563c948a 2023-02-13 00:53:55,201 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa41a4621609836c9: from storage DS-17f61d75-c836-4da5-b671-d9a9297ad34b node DatanodeRegistration(127.0.0.1:35619, datanodeUuid=8cf3b173-72eb-4183-a548-1c31563c948a, infoPort=43043, infoSecurePort=0, ipcPort=43017, storageInfo=lv=-57;cid=testClusterID;nsid=1252316509;c=1676249634767), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:55,201 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa41a4621609836c9: Processing first storage report for DS-634d7632-644c-4656-8e7e-51285747854f from datanode 8cf3b173-72eb-4183-a548-1c31563c948a 2023-02-13 00:53:55,202 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa41a4621609836c9: from storage DS-634d7632-644c-4656-8e7e-51285747854f node DatanodeRegistration(127.0.0.1:35619, datanodeUuid=8cf3b173-72eb-4183-a548-1c31563c948a, infoPort=43043, infoSecurePort=0, ipcPort=43017, storageInfo=lv=-57;cid=testClusterID;nsid=1252316509;c=1676249634767), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:55,227 INFO [Listener at localhost/43017] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36457 2023-02-13 00:53:55,236 WARN [Listener at localhost/42725] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 00:53:55,256 WARN [Listener at localhost/42725] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 00:53:55,259 WARN [Listener at localhost/42725] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 00:53:55,260 INFO [Listener at localhost/42725] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 00:53:55,267 INFO [Listener at localhost/42725] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/java.io.tmpdir/Jetty_localhost_44343_datanode____65tn4p/webapp 2023-02-13 00:53:55,335 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x99dceae3fcb5b68d: Processing first storage report for DS-f5060a68-6f1a-4840-a502-240687ab4501 from datanode 378f2e60-c4c4-4671-aa81-d1af68de1789 2023-02-13 00:53:55,335 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x99dceae3fcb5b68d: from storage DS-f5060a68-6f1a-4840-a502-240687ab4501 node DatanodeRegistration(127.0.0.1:36655, datanodeUuid=378f2e60-c4c4-4671-aa81-d1af68de1789, infoPort=42905, infoSecurePort=0, ipcPort=42725, storageInfo=lv=-57;cid=testClusterID;nsid=1252316509;c=1676249634767), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:55,336 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x99dceae3fcb5b68d: Processing first storage report for DS-db79cd52-bc48-48ef-adb4-b483910c70f8 from datanode 378f2e60-c4c4-4671-aa81-d1af68de1789 2023-02-13 00:53:55,336 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x99dceae3fcb5b68d: from storage DS-db79cd52-bc48-48ef-adb4-b483910c70f8 node DatanodeRegistration(127.0.0.1:36655, datanodeUuid=378f2e60-c4c4-4671-aa81-d1af68de1789, infoPort=42905, infoSecurePort=0, ipcPort=42725, storageInfo=lv=-57;cid=testClusterID;nsid=1252316509;c=1676249634767), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-13 00:53:55,370 INFO [Listener at localhost/42725] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44343 2023-02-13 00:53:55,378 WARN [Listener at localhost/35605] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 00:53:55,481 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x267612f09dd1c962: Processing first storage report for DS-5beb4e02-174a-467b-aab6-05de5e0deb30 from datanode dd8bba80-6bef-4cfb-9080-9a464876a28f 2023-02-13 00:53:55,481 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x267612f09dd1c962: from storage DS-5beb4e02-174a-467b-aab6-05de5e0deb30 node DatanodeRegistration(127.0.0.1:46265, datanodeUuid=dd8bba80-6bef-4cfb-9080-9a464876a28f, infoPort=38049, infoSecurePort=0, ipcPort=35605, storageInfo=lv=-57;cid=testClusterID;nsid=1252316509;c=1676249634767), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:55,481 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x267612f09dd1c962: Processing first storage report for DS-b8964a2a-57ae-4663-83dd-ee52ca212a05 from datanode dd8bba80-6bef-4cfb-9080-9a464876a28f 2023-02-13 00:53:55,481 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x267612f09dd1c962: from storage DS-b8964a2a-57ae-4663-83dd-ee52ca212a05 node DatanodeRegistration(127.0.0.1:46265, datanodeUuid=dd8bba80-6bef-4cfb-9080-9a464876a28f, infoPort=38049, infoSecurePort=0, ipcPort=35605, storageInfo=lv=-57;cid=testClusterID;nsid=1252316509;c=1676249634767), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 00:53:55,492 DEBUG [Listener at localhost/35605] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01 2023-02-13 00:53:55,495 INFO [Listener at localhost/35605] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456/zookeeper_0, clientPort=57021, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 00:53:55,496 INFO [Listener at localhost/35605] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57021 2023-02-13 00:53:55,496 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,498 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,521 INFO [Listener at localhost/35605] util.FSUtils(479): Created version file at hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc with version=8 2023-02-13 00:53:55,521 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/hbase-staging 2023-02-13 00:53:55,524 INFO [Listener at localhost/35605] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:53:55,524 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,524 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,525 INFO [Listener at localhost/35605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:55,525 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,525 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:55,525 INFO [Listener at localhost/35605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:55,528 INFO [Listener at localhost/35605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39911 2023-02-13 00:53:55,529 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,530 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,531 INFO [Listener at localhost/35605] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39911 connecting to ZooKeeper ensemble=127.0.0.1:57021 2023-02-13 00:53:55,540 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:399110x0, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:55,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39911-0x1004b2b846b0000 connected 2023-02-13 00:53:55,559 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:55,559 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:55,560 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:55,560 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39911 2023-02-13 00:53:55,560 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39911 2023-02-13 00:53:55,560 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39911 2023-02-13 00:53:55,561 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39911 2023-02-13 00:53:55,561 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39911 2023-02-13 00:53:55,561 INFO [Listener at localhost/35605] master.HMaster(439): hbase.rootdir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc, hbase.cluster.distributed=false 2023-02-13 00:53:55,575 INFO [Listener at localhost/35605] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:53:55,575 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,575 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,575 INFO [Listener at localhost/35605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:55,575 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,576 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:55,576 INFO [Listener at localhost/35605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:55,576 INFO [Listener at localhost/35605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38485 2023-02-13 00:53:55,577 INFO [Listener at localhost/35605] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:53:55,578 DEBUG [Listener at localhost/35605] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:53:55,578 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,579 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,581 INFO [Listener at localhost/35605] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38485 connecting to ZooKeeper ensemble=127.0.0.1:57021 2023-02-13 00:53:55,584 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:384850x0, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:55,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38485-0x1004b2b846b0001 connected 2023-02-13 00:53:55,586 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:55,586 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:55,587 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:55,588 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38485 2023-02-13 00:53:55,588 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38485 2023-02-13 00:53:55,588 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38485 2023-02-13 00:53:55,589 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38485 2023-02-13 00:53:55,589 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38485 2023-02-13 00:53:55,601 INFO [Listener at localhost/35605] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:53:55,601 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,602 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,602 INFO [Listener at localhost/35605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:55,602 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,602 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:55,602 INFO [Listener at localhost/35605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:55,603 INFO [Listener at localhost/35605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42839 2023-02-13 00:53:55,603 INFO [Listener at localhost/35605] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:53:55,604 DEBUG [Listener at localhost/35605] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:53:55,605 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,606 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,607 INFO [Listener at localhost/35605] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42839 connecting to ZooKeeper ensemble=127.0.0.1:57021 2023-02-13 00:53:55,610 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:428390x0, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:55,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42839-0x1004b2b846b0002 connected 2023-02-13 00:53:55,612 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:55,612 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:55,613 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:55,613 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42839 2023-02-13 00:53:55,613 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42839 2023-02-13 00:53:55,614 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42839 2023-02-13 00:53:55,614 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42839 2023-02-13 00:53:55,614 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42839 2023-02-13 00:53:55,626 INFO [Listener at localhost/35605] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:53:55,626 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,626 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,626 INFO [Listener at localhost/35605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:53:55,626 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:53:55,626 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:53:55,627 INFO [Listener at localhost/35605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:53:55,627 INFO [Listener at localhost/35605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37023 2023-02-13 00:53:55,628 INFO [Listener at localhost/35605] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:53:55,629 DEBUG [Listener at localhost/35605] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:53:55,629 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,630 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,631 INFO [Listener at localhost/35605] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37023 connecting to ZooKeeper ensemble=127.0.0.1:57021 2023-02-13 00:53:55,635 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:370230x0, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:53:55,636 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:370230x0, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:55,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37023-0x1004b2b846b0003 connected 2023-02-13 00:53:55,637 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:53:55,637 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:53:55,637 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37023 2023-02-13 00:53:55,637 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37023 2023-02-13 00:53:55,638 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37023 2023-02-13 00:53:55,638 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37023 2023-02-13 00:53:55,638 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37023 2023-02-13 00:53:55,639 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39911,1676249635523 2023-02-13 00:53:55,642 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 00:53:55,642 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39911,1676249635523 2023-02-13 00:53:55,644 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:55,644 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:55,644 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:55,644 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:55,644 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:55,645 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:53:55,647 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39911,1676249635523 from backup master directory 2023-02-13 00:53:55,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:53:55,649 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39911,1676249635523 2023-02-13 00:53:55,649 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 00:53:55,649 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:55,649 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,39911,1676249635523 2023-02-13 00:53:55,671 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/hbase.id with ID: dfbf218d-5af2-44f7-8fcf-91e5be54a5ab 2023-02-13 00:53:55,686 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:53:55,692 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:55,705 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7af2b6a1 to 127.0.0.1:57021 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:55,709 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d1f8cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:55,709 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:55,710 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 00:53:55,710 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:55,712 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store-tmp 2023-02-13 00:53:55,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:55,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 00:53:55,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:55,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:55,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 00:53:55,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:55,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:55,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676249635723Disabling compacts and flushes for region at 1676249635723Disabling writes for close at 1676249635723Writing region close event to WAL at 1676249635723Closed at 1676249635723 2023-02-13 00:53:55,724 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,39911,1676249635523 2023-02-13 00:53:55,727 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39911%2C1676249635523, suffix=, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,39911,1676249635523, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/oldWALs, maxLogs=10 2023-02-13 00:53:55,743 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:53:55,744 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:53:55,744 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:53:55,747 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,39911,1676249635523/jenkins-hbase4.apache.org%2C39911%2C1676249635523.1676249635727 2023-02-13 00:53:55,747 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK], DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK], DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK]] 2023-02-13 00:53:55,747 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:55,747 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:55,747 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:55,747 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:55,750 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:55,752 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 00:53:55,753 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 00:53:55,753 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:55,754 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:55,755 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:55,757 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:53:55,762 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:55,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66585208, jitterRate=-0.007803082466125488}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:55,762 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676249635747Initializing all the Stores at 1676249635749 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249635749Cleaning up temporary data from old regions at 1676249635755 (+6 ms)Cleaning up detritus from prior splits at 1676249635755Region opened successfully at 1676249635762 (+7 ms) 2023-02-13 00:53:55,763 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 00:53:55,764 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 00:53:55,764 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 00:53:55,764 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 00:53:55,765 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 00:53:55,765 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-02-13 00:53:55,765 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 00:53:55,766 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 00:53:55,767 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 00:53:55,779 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 00:53:55,779 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 00:53:55,780 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 00:53:55,780 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 00:53:55,780 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 00:53:55,783 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:55,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 00:53:55,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 00:53:55,784 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 00:53:55,786 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:55,786 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:55,786 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:55,786 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 00:53:55,786 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:55,786 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,39911,1676249635523, sessionid=0x1004b2b846b0000, setting cluster-up flag (Was=false) 2023-02-13 00:53:55,792 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:55,798 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 00:53:55,799 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39911,1676249635523 2023-02-13 00:53:55,801 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:55,806 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 00:53:55,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39911,1676249635523 2023-02-13 00:53:55,808 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.hbase-snapshot/.tmp 2023-02-13 00:53:55,810 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 00:53:55,810 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:55,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:55,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:55,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:53:55,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 00:53:55,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:55,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676249665814 2023-02-13 00:53:55,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 00:53:55,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 00:53:55,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 00:53:55,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 00:53:55,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 00:53:55,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 00:53:55,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,816 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 00:53:55,816 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 00:53:55,816 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 00:53:55,816 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 00:53:55,816 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 00:53:55,818 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:55,818 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 00:53:55,818 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 00:53:55,818 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249635818,5,FailOnTimeoutGroup] 2023-02-13 00:53:55,819 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249635819,5,FailOnTimeoutGroup] 2023-02-13 00:53:55,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 00:53:55,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,832 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:55,833 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:55,833 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc 2023-02-13 00:53:55,841 INFO [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(952): ClusterId : dfbf218d-5af2-44f7-8fcf-91e5be54a5ab 2023-02-13 00:53:55,842 DEBUG [RS:0;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:53:55,842 INFO [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(952): ClusterId : dfbf218d-5af2-44f7-8fcf-91e5be54a5ab 2023-02-13 00:53:55,842 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(952): ClusterId : dfbf218d-5af2-44f7-8fcf-91e5be54a5ab 2023-02-13 00:53:55,842 DEBUG [RS:2;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:53:55,843 DEBUG [RS:1;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:53:55,845 DEBUG [RS:0;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:53:55,845 DEBUG [RS:0;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:53:55,845 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:55,845 DEBUG [RS:2;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:53:55,845 DEBUG [RS:2;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:53:55,846 DEBUG [RS:1;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:53:55,846 DEBUG [RS:1;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:53:55,849 DEBUG [RS:0;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:53:55,849 DEBUG [RS:1;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:53:55,849 DEBUG [RS:2;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:53:55,850 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 00:53:55,853 DEBUG [RS:1;jenkins-hbase4:42839] zookeeper.ReadOnlyZKClient(139): Connect 0x66110ead to 127.0.0.1:57021 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:55,853 DEBUG [RS:0;jenkins-hbase4:38485] zookeeper.ReadOnlyZKClient(139): Connect 0x53616874 to 127.0.0.1:57021 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:55,853 DEBUG [RS:2;jenkins-hbase4:37023] zookeeper.ReadOnlyZKClient(139): Connect 0x75ae988a to 127.0.0.1:57021 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:55,853 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info 2023-02-13 00:53:55,854 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 00:53:55,855 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:55,855 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 00:53:55,867 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:53:55,867 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 00:53:55,868 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:55,868 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 00:53:55,869 DEBUG [RS:0;jenkins-hbase4:38485] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5da68c68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:55,870 DEBUG [RS:0;jenkins-hbase4:38485] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c9b8653, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:55,870 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/table 2023-02-13 00:53:55,870 DEBUG [RS:2;jenkins-hbase4:37023] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f56f3fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:55,870 DEBUG [RS:1;jenkins-hbase4:42839] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ce965dd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:55,870 DEBUG [RS:2;jenkins-hbase4:37023] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a547274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:55,870 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 00:53:55,870 DEBUG [RS:1;jenkins-hbase4:42839] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29ebf624, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:55,872 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:55,874 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740 2023-02-13 00:53:55,875 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740 2023-02-13 00:53:55,878 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 00:53:55,880 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 00:53:55,880 DEBUG [RS:0;jenkins-hbase4:38485] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:38485 2023-02-13 00:53:55,880 INFO [RS:0;jenkins-hbase4:38485] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:53:55,881 INFO [RS:0;jenkins-hbase4:38485] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:53:55,881 DEBUG [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:53:55,881 INFO [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39911,1676249635523 with isa=jenkins-hbase4.apache.org/172.31.14.131:38485, startcode=1676249635575 2023-02-13 00:53:55,881 DEBUG [RS:0;jenkins-hbase4:38485] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:53:55,882 DEBUG [RS:1;jenkins-hbase4:42839] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:42839 2023-02-13 00:53:55,882 INFO [RS:1;jenkins-hbase4:42839] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:53:55,882 INFO [RS:1;jenkins-hbase4:42839] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:53:55,882 DEBUG [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:53:55,883 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39911,1676249635523 with isa=jenkins-hbase4.apache.org/172.31.14.131:42839, startcode=1676249635601 2023-02-13 00:53:55,883 DEBUG [RS:1;jenkins-hbase4:42839] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:53:55,883 DEBUG [RS:2;jenkins-hbase4:37023] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:37023 2023-02-13 00:53:55,883 INFO [RS:2;jenkins-hbase4:37023] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:53:55,883 INFO [RS:2;jenkins-hbase4:37023] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:53:55,883 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:55,883 DEBUG [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:53:55,883 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43049, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:53:55,884 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60220988, jitterRate=-0.10263735055923462}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 00:53:55,885 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676249635845Initializing all the Stores at 1676249635847 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249635847Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249635850 (+3 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249635850Cleaning up temporary data from old regions at 1676249635877 (+27 ms)Cleaning up detritus from prior splits at 1676249635877Region opened successfully at 1676249635885 (+8 ms) 2023-02-13 00:53:55,885 INFO [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39911,1676249635523 with isa=jenkins-hbase4.apache.org/172.31.14.131:37023, startcode=1676249635625 2023-02-13 00:53:55,885 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 00:53:55,885 DEBUG [RS:2;jenkins-hbase4:37023] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:53:55,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:55,886 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39135, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.6 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:53:55,886 DEBUG [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc 2023-02-13 00:53:55,886 DEBUG [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44257 2023-02-13 00:53:55,886 DEBUG [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:53:55,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:55,885 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 00:53:55,887 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 00:53:55,887 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 00:53:55,887 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 00:53:55,887 DEBUG [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc 2023-02-13 00:53:55,887 DEBUG [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44257 2023-02-13 00:53:55,887 INFO [RS-EventLoopGroup-9-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54757, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.7 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:53:55,887 DEBUG [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:53:55,888 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 00:53:55,888 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676249635885Disabling compacts and flushes for region at 1676249635885Disabling writes for close at 1676249635887 (+2 ms)Writing region close event to WAL at 1676249635888 (+1 ms)Closed at 1676249635888 2023-02-13 00:53:55,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:55,888 DEBUG [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc 2023-02-13 00:53:55,888 DEBUG [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44257 2023-02-13 00:53:55,888 DEBUG [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:53:55,889 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:55,889 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 00:53:55,889 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 00:53:55,889 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 00:53:55,893 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 00:53:55,896 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 00:53:55,897 DEBUG [RS:0;jenkins-hbase4:38485] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:55,897 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37023,1676249635625] 2023-02-13 00:53:55,897 WARN [RS:0;jenkins-hbase4:38485] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:55,897 DEBUG [RS:1;jenkins-hbase4:42839] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:55,897 INFO [RS:0;jenkins-hbase4:38485] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:55,897 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38485,1676249635575] 2023-02-13 00:53:55,897 WARN [RS:1;jenkins-hbase4:42839] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:55,897 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42839,1676249635601] 2023-02-13 00:53:55,897 DEBUG [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:55,898 INFO [RS:1;jenkins-hbase4:42839] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:55,898 DEBUG [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:55,900 DEBUG [RS:2;jenkins-hbase4:37023] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:55,900 WARN [RS:2;jenkins-hbase4:37023] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:53:55,900 INFO [RS:2;jenkins-hbase4:37023] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:55,900 DEBUG [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:55,903 DEBUG [RS:0;jenkins-hbase4:38485] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:55,903 DEBUG [RS:1;jenkins-hbase4:42839] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:55,904 DEBUG [RS:1;jenkins-hbase4:42839] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:55,904 DEBUG [RS:0;jenkins-hbase4:38485] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:55,904 DEBUG [RS:1;jenkins-hbase4:42839] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:55,905 DEBUG [RS:0;jenkins-hbase4:38485] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:55,905 DEBUG [RS:2;jenkins-hbase4:37023] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:55,905 DEBUG [RS:1;jenkins-hbase4:42839] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:53:55,905 DEBUG [RS:2;jenkins-hbase4:37023] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:55,906 INFO [RS:1;jenkins-hbase4:42839] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:53:55,906 DEBUG [RS:0;jenkins-hbase4:38485] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:53:55,906 DEBUG [RS:2;jenkins-hbase4:37023] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:55,907 DEBUG [RS:2;jenkins-hbase4:37023] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:53:55,908 INFO [RS:2;jenkins-hbase4:37023] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:53:55,908 INFO [RS:1;jenkins-hbase4:42839] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:53:55,910 INFO [RS:0;jenkins-hbase4:38485] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:53:55,912 INFO [RS:1;jenkins-hbase4:42839] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:53:55,912 INFO [RS:1;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,912 INFO [RS:0;jenkins-hbase4:38485] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:53:55,912 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:53:55,912 INFO [RS:2;jenkins-hbase4:37023] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:53:55,916 INFO [RS:1;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,916 INFO [RS:0;jenkins-hbase4:38485] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:53:55,916 INFO [RS:2;jenkins-hbase4:37023] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:53:55,916 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,917 INFO [RS:2;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,916 INFO [RS:0;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,917 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,917 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,917 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,917 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,917 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:55,917 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,917 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,917 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,917 DEBUG [RS:1;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,922 INFO [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:53:55,924 INFO [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:53:55,925 INFO [RS:1;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,925 INFO [RS:1;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,925 INFO [RS:1;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,928 INFO [RS:0;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,928 INFO [RS:2;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,928 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,928 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,928 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,928 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,928 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,928 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,928 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,928 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:55,929 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:53:55,929 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:0;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,929 DEBUG [RS:2;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:53:55,932 INFO [RS:0;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,932 INFO [RS:0;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,933 INFO [RS:0;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,933 INFO [RS:2;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,933 INFO [RS:2;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,933 INFO [RS:2;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,947 INFO [RS:0;jenkins-hbase4:38485] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:53:55,947 INFO [RS:0;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38485,1676249635575-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,947 INFO [RS:2;jenkins-hbase4:37023] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:53:55,947 INFO [RS:2;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37023,1676249635625-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:55,948 INFO [RS:1;jenkins-hbase4:42839] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:53:55,948 INFO [RS:1;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42839,1676249635601-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:56,010 INFO [RS:2;jenkins-hbase4:37023] regionserver.Replication(203): jenkins-hbase4.apache.org,37023,1676249635625 started 2023-02-13 00:53:56,010 INFO [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,37023,1676249635625, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37023, sessionid=0x1004b2b846b0003 2023-02-13 00:53:56,010 DEBUG [RS:2;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:53:56,011 DEBUG [RS:2;jenkins-hbase4:37023] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:56,011 DEBUG [RS:2;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37023,1676249635625' 2023-02-13 00:53:56,011 DEBUG [RS:2;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:53:56,011 DEBUG [RS:2;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:53:56,012 DEBUG [RS:2;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:53:56,012 DEBUG [RS:2;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:53:56,012 DEBUG [RS:2;jenkins-hbase4:37023] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:56,012 DEBUG [RS:2;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37023,1676249635625' 2023-02-13 00:53:56,012 DEBUG [RS:2;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:53:56,012 DEBUG [RS:2;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:53:56,012 DEBUG [RS:2;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:53:56,012 INFO [RS:2;jenkins-hbase4:37023] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:53:56,013 INFO [RS:2;jenkins-hbase4:37023] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:53:56,014 INFO [RS:0;jenkins-hbase4:38485] regionserver.Replication(203): jenkins-hbase4.apache.org,38485,1676249635575 started 2023-02-13 00:53:56,014 INFO [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38485,1676249635575, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38485, sessionid=0x1004b2b846b0001 2023-02-13 00:53:56,014 DEBUG [RS:0;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:53:56,014 DEBUG [RS:0;jenkins-hbase4:38485] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:56,014 DEBUG [RS:0;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38485,1676249635575' 2023-02-13 00:53:56,014 DEBUG [RS:0;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:53:56,015 DEBUG [RS:0;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:53:56,015 DEBUG [RS:0;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:53:56,015 DEBUG [RS:0;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:53:56,015 DEBUG [RS:0;jenkins-hbase4:38485] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:56,015 DEBUG [RS:0;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38485,1676249635575' 2023-02-13 00:53:56,015 DEBUG [RS:0;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:53:56,015 DEBUG [RS:0;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:53:56,016 DEBUG [RS:0;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:53:56,016 INFO [RS:0;jenkins-hbase4:38485] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:53:56,016 INFO [RS:0;jenkins-hbase4:38485] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:53:56,020 INFO [RS:1;jenkins-hbase4:42839] regionserver.Replication(203): jenkins-hbase4.apache.org,42839,1676249635601 started 2023-02-13 00:53:56,020 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42839,1676249635601, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42839, sessionid=0x1004b2b846b0002 2023-02-13 00:53:56,020 DEBUG [RS:1;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:53:56,020 DEBUG [RS:1;jenkins-hbase4:42839] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:56,020 DEBUG [RS:1;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42839,1676249635601' 2023-02-13 00:53:56,020 DEBUG [RS:1;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:53:56,020 DEBUG [RS:1;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:53:56,021 DEBUG [RS:1;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:53:56,021 DEBUG [RS:1;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:53:56,021 DEBUG [RS:1;jenkins-hbase4:42839] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:56,021 DEBUG [RS:1;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42839,1676249635601' 2023-02-13 00:53:56,021 DEBUG [RS:1;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:53:56,021 DEBUG [RS:1;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:53:56,022 DEBUG [RS:1;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:53:56,022 INFO [RS:1;jenkins-hbase4:42839] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:53:56,022 INFO [RS:1;jenkins-hbase4:42839] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:53:56,046 DEBUG [jenkins-hbase4:39911] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 00:53:56,046 DEBUG [jenkins-hbase4:39911] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:56,055 DEBUG [jenkins-hbase4:39911] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:56,055 DEBUG [jenkins-hbase4:39911] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:56,055 DEBUG [jenkins-hbase4:39911] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:56,055 DEBUG [jenkins-hbase4:39911] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:56,056 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42839,1676249635601, state=OPENING 2023-02-13 00:53:56,058 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 00:53:56,064 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:56,065 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 00:53:56,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42839,1676249635601}] 2023-02-13 00:53:56,115 INFO [RS:2;jenkins-hbase4:37023] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37023%2C1676249635625, suffix=, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249635625, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs, maxLogs=32 2023-02-13 00:53:56,117 INFO [RS:0;jenkins-hbase4:38485] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38485%2C1676249635575, suffix=, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249635575, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs, maxLogs=32 2023-02-13 00:53:56,123 INFO [RS:1;jenkins-hbase4:42839] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42839%2C1676249635601, suffix=, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs, maxLogs=32 2023-02-13 00:53:56,144 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:53:56,144 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:53:56,144 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:53:56,150 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:53:56,150 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:53:56,150 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:53:56,152 INFO [RS:2;jenkins-hbase4:37023] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249635625/jenkins-hbase4.apache.org%2C37023%2C1676249635625.1676249636116 2023-02-13 00:53:56,154 INFO [RS:0;jenkins-hbase4:38485] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249635575/jenkins-hbase4.apache.org%2C38485%2C1676249635575.1676249636118 2023-02-13 00:53:56,154 DEBUG [RS:2;jenkins-hbase4:37023] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK], DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK], DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK]] 2023-02-13 00:53:56,154 DEBUG [RS:0;jenkins-hbase4:38485] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK], DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK], DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK]] 2023-02-13 00:53:56,156 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:53:56,156 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:53:56,156 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:53:56,164 INFO [RS:1;jenkins-hbase4:42839] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601/jenkins-hbase4.apache.org%2C42839%2C1676249635601.1676249636124 2023-02-13 00:53:56,164 DEBUG [RS:1;jenkins-hbase4:42839] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK], DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK], DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK]] 2023-02-13 00:53:56,220 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:56,220 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 00:53:56,222 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37008, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 00:53:56,226 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 00:53:56,226 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:53:56,228 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42839%2C1676249635601.meta, suffix=.meta, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs, maxLogs=32 2023-02-13 00:53:56,242 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:53:56,242 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:53:56,242 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:53:56,244 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601/jenkins-hbase4.apache.org%2C42839%2C1676249635601.meta.1676249636228.meta 2023-02-13 00:53:56,244 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK], DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK], DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK]] 2023-02-13 00:53:56,244 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:56,244 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 00:53:56,245 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 00:53:56,245 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 00:53:56,245 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 00:53:56,245 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:56,245 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 00:53:56,245 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 00:53:56,246 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 00:53:56,247 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info 2023-02-13 00:53:56,247 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info 2023-02-13 00:53:56,248 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 00:53:56,248 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:56,249 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 00:53:56,249 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:53:56,249 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:53:56,250 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 00:53:56,250 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:56,251 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 00:53:56,251 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/table 2023-02-13 00:53:56,251 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/table 2023-02-13 00:53:56,252 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 00:53:56,252 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:56,253 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740 2023-02-13 00:53:56,254 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740 2023-02-13 00:53:56,257 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 00:53:56,259 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 00:53:56,260 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72571148, jitterRate=0.08139437437057495}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 00:53:56,261 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676249636245Writing region info on filesystem at 1676249636245Initializing all the Stores at 1676249636246 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249636246Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249636246Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249636246Cleaning up temporary data from old regions at 1676249636256 (+10 ms)Cleaning up detritus from prior splits at 1676249636256Running coprocessor post-open hooks at 1676249636260 (+4 ms)Region opened successfully at 1676249636260 2023-02-13 00:53:56,265 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676249636220 2023-02-13 00:53:56,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 00:53:56,271 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 00:53:56,272 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42839,1676249635601, state=OPEN 2023-02-13 00:53:56,275 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 00:53:56,275 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 00:53:56,278 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 00:53:56,278 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42839,1676249635601 in 210 msec 2023-02-13 00:53:56,280 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 00:53:56,280 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 389 msec 2023-02-13 00:53:56,283 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 472 msec 2023-02-13 00:53:56,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676249635642, completionTime=-1 2023-02-13 00:53:56,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 00:53:56,283 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 00:53:56,286 DEBUG [hconnection-0x2e1da87-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 00:53:56,287 INFO [RS-EventLoopGroup-11-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37024, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 00:53:56,289 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 00:53:56,289 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676249696289 2023-02-13 00:53:56,289 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676249756289 2023-02-13 00:53:56,289 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 6 msec 2023-02-13 00:53:56,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39911,1676249635523-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:56,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39911,1676249635523-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:56,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39911,1676249635523-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:56,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39911, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:56,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 00:53:56,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 00:53:56,297 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:56,298 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 00:53:56,298 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 00:53:56,299 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 00:53:56,300 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 00:53:56,302 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:56,302 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f empty. 2023-02-13 00:53:56,303 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:56,303 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 00:53:56,321 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:56,322 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => e95173dbc5ee3e5a46e6e44669d98b1f, NAME => 'hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp 2023-02-13 00:53:56,334 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:56,334 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing e95173dbc5ee3e5a46e6e44669d98b1f, disabling compactions & flushes 2023-02-13 00:53:56,334 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:56,334 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:56,334 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. after waiting 0 ms 2023-02-13 00:53:56,334 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:56,334 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:56,334 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for e95173dbc5ee3e5a46e6e44669d98b1f: Waiting for close lock at 1676249636334Disabling compacts and flushes for region at 1676249636334Disabling writes for close at 1676249636334Writing region close event to WAL at 1676249636334Closed at 1676249636334 2023-02-13 00:53:56,336 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 00:53:56,337 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249636337"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676249636337"}]},"ts":"1676249636337"} 2023-02-13 00:53:56,339 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 00:53:56,340 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 00:53:56,341 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249636340"}]},"ts":"1676249636340"} 2023-02-13 00:53:56,342 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 00:53:56,345 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:56,346 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:56,346 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:56,346 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:56,346 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:56,346 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e95173dbc5ee3e5a46e6e44669d98b1f, ASSIGN}] 2023-02-13 00:53:56,349 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e95173dbc5ee3e5a46e6e44669d98b1f, ASSIGN 2023-02-13 00:53:56,349 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e95173dbc5ee3e5a46e6e44669d98b1f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42839,1676249635601; forceNewPlan=false, retain=false 2023-02-13 00:53:56,500 INFO [jenkins-hbase4:39911] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 00:53:56,501 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e95173dbc5ee3e5a46e6e44669d98b1f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:56,501 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249636501"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249636501"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249636501"}]},"ts":"1676249636501"} 2023-02-13 00:53:56,503 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e95173dbc5ee3e5a46e6e44669d98b1f, server=jenkins-hbase4.apache.org,42839,1676249635601}] 2023-02-13 00:53:56,555 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-13 00:53:56,660 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:56,660 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e95173dbc5ee3e5a46e6e44669d98b1f, NAME => 'hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:56,660 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:56,661 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:56,661 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:56,661 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:56,662 INFO [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:56,664 DEBUG [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/info 2023-02-13 00:53:56,664 DEBUG [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/info 2023-02-13 00:53:56,664 INFO [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e95173dbc5ee3e5a46e6e44669d98b1f columnFamilyName info 2023-02-13 00:53:56,665 INFO [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] regionserver.HStore(310): Store=e95173dbc5ee3e5a46e6e44669d98b1f/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:56,665 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:56,666 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:56,669 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:56,671 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:56,671 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e95173dbc5ee3e5a46e6e44669d98b1f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71024157, jitterRate=0.05834241211414337}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:56,671 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e95173dbc5ee3e5a46e6e44669d98b1f: Running coprocessor pre-open hook at 1676249636661Writing region info on filesystem at 1676249636661Initializing all the Stores at 1676249636662 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249636662Cleaning up temporary data from old regions at 1676249636666 (+4 ms)Cleaning up detritus from prior splits at 1676249636667 (+1 ms)Running coprocessor post-open hooks at 1676249636671 (+4 ms)Region opened successfully at 1676249636671 2023-02-13 00:53:56,672 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f., pid=6, masterSystemTime=1676249636656 2023-02-13 00:53:56,674 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:56,674 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:56,675 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e95173dbc5ee3e5a46e6e44669d98b1f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:56,675 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249636675"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249636675"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249636675"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249636675"}]},"ts":"1676249636675"} 2023-02-13 00:53:56,679 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 00:53:56,679 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e95173dbc5ee3e5a46e6e44669d98b1f, server=jenkins-hbase4.apache.org,42839,1676249635601 in 174 msec 2023-02-13 00:53:56,681 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 00:53:56,681 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e95173dbc5ee3e5a46e6e44669d98b1f, ASSIGN in 333 msec 2023-02-13 00:53:56,682 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 00:53:56,682 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249636682"}]},"ts":"1676249636682"} 2023-02-13 00:53:56,683 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 00:53:56,686 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 00:53:56,688 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 389 msec 2023-02-13 00:53:56,699 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 00:53:56,701 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 00:53:56,701 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:56,705 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 00:53:56,716 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 00:53:56,719 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 13 msec 2023-02-13 00:53:56,729 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 00:53:56,741 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 00:53:56,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-02-13 00:53:56,754 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 00:53:56,758 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 00:53:56,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 1.109sec 2023-02-13 00:53:56,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 00:53:56,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 00:53:56,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 00:53:56,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39911,1676249635523-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 00:53:56,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39911,1676249635523-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 00:53:56,761 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 00:53:56,850 DEBUG [Listener at localhost/35605] zookeeper.ReadOnlyZKClient(139): Connect 0x7ea9bbe5 to 127.0.0.1:57021 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:53:56,856 DEBUG [Listener at localhost/35605] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e79bd73, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:53:56,858 DEBUG [hconnection-0x429f910b-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 00:53:56,860 INFO [RS-EventLoopGroup-11-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37034, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 00:53:56,862 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39911,1676249635523 2023-02-13 00:53:56,866 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-13 00:53:56,866 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:56,867 INFO [Listener at localhost/35605] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-13 00:53:56,867 INFO [Listener at localhost/35605] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-13 00:53:56,870 DEBUG [Listener at localhost/35605] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 00:53:56,872 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44752, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 00:53:56,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:56,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-13 00:53:56,878 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 00:53:56,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-13 00:53:56,879 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 00:53:56,881 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:56,882 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84 empty. 2023-02-13 00:53:56,882 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:56,882 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-13 00:53:56,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 00:53:56,901 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:56,902 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => e0dbd631ed14dc0143960d5a5baf5b84, NAME => 'restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp 2023-02-13 00:53:56,913 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:56,913 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing e0dbd631ed14dc0143960d5a5baf5b84, disabling compactions & flushes 2023-02-13 00:53:56,914 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:56,914 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:56,914 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. after waiting 0 ms 2023-02-13 00:53:56,914 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:56,914 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:56,914 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for e0dbd631ed14dc0143960d5a5baf5b84: Waiting for close lock at 1676249636913Disabling compacts and flushes for region at 1676249636913Disabling writes for close at 1676249636914 (+1 ms)Writing region close event to WAL at 1676249636914Closed at 1676249636914 2023-02-13 00:53:56,916 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 00:53:56,917 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249636917"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676249636917"}]},"ts":"1676249636917"} 2023-02-13 00:53:56,919 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 00:53:56,920 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 00:53:56,920 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249636920"}]},"ts":"1676249636920"} 2023-02-13 00:53:56,922 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-13 00:53:56,925 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:56,926 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:56,926 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:56,926 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:56,926 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:56,926 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=e0dbd631ed14dc0143960d5a5baf5b84, ASSIGN}] 2023-02-13 00:53:56,928 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=e0dbd631ed14dc0143960d5a5baf5b84, ASSIGN 2023-02-13 00:53:56,929 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=e0dbd631ed14dc0143960d5a5baf5b84, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42839,1676249635601; forceNewPlan=false, retain=false 2023-02-13 00:53:56,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 00:53:57,079 INFO [jenkins-hbase4:39911] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 00:53:57,081 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e0dbd631ed14dc0143960d5a5baf5b84, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:57,081 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249637081"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249637081"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249637081"}]},"ts":"1676249637081"} 2023-02-13 00:53:57,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure e0dbd631ed14dc0143960d5a5baf5b84, server=jenkins-hbase4.apache.org,42839,1676249635601}] 2023-02-13 00:53:57,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 00:53:57,239 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:57,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0dbd631ed14dc0143960d5a5baf5b84, NAME => 'restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:57,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:57,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:57,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:57,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:57,242 INFO [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:57,244 DEBUG [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84/family 2023-02-13 00:53:57,244 DEBUG [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84/family 2023-02-13 00:53:57,244 INFO [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0dbd631ed14dc0143960d5a5baf5b84 columnFamilyName family 2023-02-13 00:53:57,245 INFO [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] regionserver.HStore(310): Store=e0dbd631ed14dc0143960d5a5baf5b84/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:57,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:57,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:57,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:57,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:57,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e0dbd631ed14dc0143960d5a5baf5b84; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60963020, jitterRate=-0.09158021211624146}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:57,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e0dbd631ed14dc0143960d5a5baf5b84: Running coprocessor pre-open hook at 1676249637241Writing region info on filesystem at 1676249637241Initializing all the Stores at 1676249637242 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249637242Cleaning up temporary data from old regions at 1676249637247 (+5 ms)Cleaning up detritus from prior splits at 1676249637248 (+1 ms)Running coprocessor post-open hooks at 1676249637254 (+6 ms)Region opened successfully at 1676249637254 2023-02-13 00:53:57,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84., pid=11, masterSystemTime=1676249637235 2023-02-13 00:53:57,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:57,257 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:57,258 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e0dbd631ed14dc0143960d5a5baf5b84, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:57,258 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249637258"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249637258"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249637258"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249637258"}]},"ts":"1676249637258"} 2023-02-13 00:53:57,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-13 00:53:57,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure e0dbd631ed14dc0143960d5a5baf5b84, server=jenkins-hbase4.apache.org,42839,1676249635601 in 177 msec 2023-02-13 00:53:57,266 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-13 00:53:57,266 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=e0dbd631ed14dc0143960d5a5baf5b84, ASSIGN in 337 msec 2023-02-13 00:53:57,267 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 00:53:57,267 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249637267"}]},"ts":"1676249637267"} 2023-02-13 00:53:57,269 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-13 00:53:57,272 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 00:53:57,273 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 398 msec 2023-02-13 00:53:57,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 00:53:57,490 INFO [Listener at localhost/35605] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-13 00:53:57,490 DEBUG [Listener at localhost/35605] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-13 00:53:57,490 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:57,493 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-13 00:53:57,493 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:57,493 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-13 00:53:57,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:57,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-13 00:53:57,498 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 00:53:57,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-13 00:53:57,499 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 00:53:57,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 00:53:57,501 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed 2023-02-13 00:53:57,502 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed empty. 2023-02-13 00:53:57,502 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed 2023-02-13 00:53:57,502 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-13 00:53:57,517 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:57,518 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => 0a98b367f319e93352882eb969190eed, NAME => 'restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp 2023-02-13 00:53:57,533 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:57,533 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing 0a98b367f319e93352882eb969190eed, disabling compactions & flushes 2023-02-13 00:53:57,533 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:57,533 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:57,533 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. after waiting 0 ms 2023-02-13 00:53:57,533 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:57,533 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:57,534 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for 0a98b367f319e93352882eb969190eed: Waiting for close lock at 1676249637533Disabling compacts and flushes for region at 1676249637533Disabling writes for close at 1676249637533Writing region close event to WAL at 1676249637533Closed at 1676249637533 2023-02-13 00:53:57,537 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 00:53:57,538 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249637538"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676249637538"}]},"ts":"1676249637538"} 2023-02-13 00:53:57,540 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 00:53:57,541 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 00:53:57,541 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249637541"}]},"ts":"1676249637541"} 2023-02-13 00:53:57,543 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-13 00:53:57,548 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:57,549 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:57,549 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:57,549 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:57,549 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:57,549 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=0a98b367f319e93352882eb969190eed, ASSIGN}] 2023-02-13 00:53:57,551 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=0a98b367f319e93352882eb969190eed, ASSIGN 2023-02-13 00:53:57,552 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=0a98b367f319e93352882eb969190eed, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38485,1676249635575; forceNewPlan=false, retain=false 2023-02-13 00:53:57,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 00:53:57,702 INFO [jenkins-hbase4:39911] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 00:53:57,703 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=0a98b367f319e93352882eb969190eed, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:57,703 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249637703"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249637703"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249637703"}]},"ts":"1676249637703"} 2023-02-13 00:53:57,706 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 0a98b367f319e93352882eb969190eed, server=jenkins-hbase4.apache.org,38485,1676249635575}] 2023-02-13 00:53:57,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 00:53:57,859 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:57,859 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 00:53:57,860 INFO [RS-EventLoopGroup-10-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54662, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 00:53:57,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:57,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0a98b367f319e93352882eb969190eed, NAME => 'restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:57,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 0a98b367f319e93352882eb969190eed 2023-02-13 00:53:57,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:57,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0a98b367f319e93352882eb969190eed 2023-02-13 00:53:57,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0a98b367f319e93352882eb969190eed 2023-02-13 00:53:57,866 INFO [StoreOpener-0a98b367f319e93352882eb969190eed-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 0a98b367f319e93352882eb969190eed 2023-02-13 00:53:57,868 DEBUG [StoreOpener-0a98b367f319e93352882eb969190eed-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed/family 2023-02-13 00:53:57,868 DEBUG [StoreOpener-0a98b367f319e93352882eb969190eed-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed/family 2023-02-13 00:53:57,868 INFO [StoreOpener-0a98b367f319e93352882eb969190eed-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a98b367f319e93352882eb969190eed columnFamilyName family 2023-02-13 00:53:57,869 INFO [StoreOpener-0a98b367f319e93352882eb969190eed-1] regionserver.HStore(310): Store=0a98b367f319e93352882eb969190eed/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:57,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed 2023-02-13 00:53:57,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed 2023-02-13 00:53:57,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0a98b367f319e93352882eb969190eed 2023-02-13 00:53:57,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:57,877 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0a98b367f319e93352882eb969190eed; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65476427, jitterRate=-0.024325206875801086}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:57,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0a98b367f319e93352882eb969190eed: Running coprocessor pre-open hook at 1676249637865Writing region info on filesystem at 1676249637865Initializing all the Stores at 1676249637866 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249637866Cleaning up temporary data from old regions at 1676249637870 (+4 ms)Cleaning up detritus from prior splits at 1676249637871 (+1 ms)Running coprocessor post-open hooks at 1676249637877 (+6 ms)Region opened successfully at 1676249637877 2023-02-13 00:53:57,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed., pid=14, masterSystemTime=1676249637858 2023-02-13 00:53:57,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:57,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:57,883 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=0a98b367f319e93352882eb969190eed, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:57,883 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249637882"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249637882"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249637882"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249637882"}]},"ts":"1676249637882"} 2023-02-13 00:53:57,887 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-13 00:53:57,887 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 0a98b367f319e93352882eb969190eed, server=jenkins-hbase4.apache.org,38485,1676249635575 in 179 msec 2023-02-13 00:53:57,890 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-13 00:53:57,890 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=0a98b367f319e93352882eb969190eed, ASSIGN in 338 msec 2023-02-13 00:53:57,891 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 00:53:57,891 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249637891"}]},"ts":"1676249637891"} 2023-02-13 00:53:57,892 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-13 00:53:57,895 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 00:53:57,896 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 399 msec 2023-02-13 00:53:58,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 00:53:58,104 INFO [Listener at localhost/35605] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-13 00:53:58,104 DEBUG [Listener at localhost/35605] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-13 00:53:58,104 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:58,107 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-13 00:53:58,107 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:58,107 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-13 00:53:58,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:53:58,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-13 00:53:58,113 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 00:53:58,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-13 00:53:58,114 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 00:53:58,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 00:53:58,116 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,116 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95 empty. 2023-02-13 00:53:58,117 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,117 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-13 00:53:58,142 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-13 00:53:58,144 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => 4534c163be465f052ed9a3af03edbb95, NAME => 'restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.tmp 2023-02-13 00:53:58,155 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:58,155 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing 4534c163be465f052ed9a3af03edbb95, disabling compactions & flushes 2023-02-13 00:53:58,155 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,155 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,155 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. after waiting 0 ms 2023-02-13 00:53:58,155 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,155 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,155 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for 4534c163be465f052ed9a3af03edbb95: Waiting for close lock at 1676249638155Disabling compacts and flushes for region at 1676249638155Disabling writes for close at 1676249638155Writing region close event to WAL at 1676249638155Closed at 1676249638155 2023-02-13 00:53:58,158 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 00:53:58,159 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676249638159"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676249638159"}]},"ts":"1676249638159"} 2023-02-13 00:53:58,161 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 00:53:58,168 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 00:53:58,168 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249638168"}]},"ts":"1676249638168"} 2023-02-13 00:53:58,170 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-13 00:53:58,173 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 00:53:58,174 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 00:53:58,174 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 00:53:58,174 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 00:53:58,174 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 00:53:58,174 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=4534c163be465f052ed9a3af03edbb95, ASSIGN}] 2023-02-13 00:53:58,176 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=4534c163be465f052ed9a3af03edbb95, ASSIGN 2023-02-13 00:53:58,178 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=4534c163be465f052ed9a3af03edbb95, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42839,1676249635601; forceNewPlan=false, retain=false 2023-02-13 00:53:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 00:53:58,328 INFO [jenkins-hbase4:39911] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 00:53:58,329 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=4534c163be465f052ed9a3af03edbb95, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:58,330 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676249638329"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249638329"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249638329"}]},"ts":"1676249638329"} 2023-02-13 00:53:58,334 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 4534c163be465f052ed9a3af03edbb95, server=jenkins-hbase4.apache.org,42839,1676249635601}] 2023-02-13 00:53:58,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 00:53:58,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4534c163be465f052ed9a3af03edbb95, NAME => 'restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:53:58,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:53:58,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,499 INFO [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,504 DEBUG [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95/family 2023-02-13 00:53:58,504 DEBUG [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95/family 2023-02-13 00:53:58,505 INFO [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4534c163be465f052ed9a3af03edbb95 columnFamilyName family 2023-02-13 00:53:58,507 INFO [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] regionserver.HStore(310): Store=4534c163be465f052ed9a3af03edbb95/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:53:58,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,513 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 00:53:58,517 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4534c163be465f052ed9a3af03edbb95; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70734748, jitterRate=0.054029881954193115}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:53:58,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4534c163be465f052ed9a3af03edbb95: Running coprocessor pre-open hook at 1676249638494Writing region info on filesystem at 1676249638494Initializing all the Stores at 1676249638498 (+4 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249638498Cleaning up temporary data from old regions at 1676249638509 (+11 ms)Cleaning up detritus from prior splits at 1676249638510 (+1 ms)Running coprocessor post-open hooks at 1676249638517 (+7 ms)Region opened successfully at 1676249638517 2023-02-13 00:53:58,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95., pid=17, masterSystemTime=1676249638489 2023-02-13 00:53:58,522 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=4534c163be465f052ed9a3af03edbb95, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:58,522 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676249638522"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249638522"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249638522"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249638522"}]},"ts":"1676249638522"} 2023-02-13 00:53:58,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,523 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,526 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-13 00:53:58,527 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 4534c163be465f052ed9a3af03edbb95, server=jenkins-hbase4.apache.org,42839,1676249635601 in 190 msec 2023-02-13 00:53:58,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-13 00:53:58,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=4534c163be465f052ed9a3af03edbb95, ASSIGN in 353 msec 2023-02-13 00:53:58,530 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 00:53:58,530 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676249638530"}]},"ts":"1676249638530"} 2023-02-13 00:53:58,532 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-13 00:53:58,534 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 00:53:58,538 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 425 msec 2023-02-13 00:53:58,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39911] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 00:53:58,719 INFO [Listener at localhost/35605] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-13 00:53:58,719 DEBUG [Listener at localhost/35605] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-13 00:53:58,719 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:58,721 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-13 00:53:58,722 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:58,722 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-13 00:53:58,722 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:58,725 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:58,727 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:58,730 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:53:58,730 INFO [Listener at localhost/35605] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 00:53:58,733 INFO [Listener at localhost/35605] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 00:53:58,733 INFO [Listener at localhost/35605] master.TestRetainAssignmentOnRestart(88): Shutting down HBase cluster 2023-02-13 00:53:58,733 INFO [Listener at localhost/35605] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase4:39911,5,FailOnTimeoutGroup] 2023-02-13 00:53:58,733 INFO [Listener at localhost/35605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39911,1676249635523' ***** 2023-02-13 00:53:58,733 INFO [Listener at localhost/35605] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-13 00:53:58,734 DEBUG [M:0;jenkins-hbase4:39911] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6488294c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:53:58,734 INFO [M:0;jenkins-hbase4:39911] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39911,1676249635523 2023-02-13 00:53:58,735 DEBUG [M:0;jenkins-hbase4:39911] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7af2b6a1 to 127.0.0.1:57021 2023-02-13 00:53:58,735 DEBUG [M:0;jenkins-hbase4:39911] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:58,735 INFO [M:0;jenkins-hbase4:39911] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39911,1676249635523; all regions closed. 2023-02-13 00:53:58,735 DEBUG [M:0;jenkins-hbase4:39911] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:58,735 DEBUG [M:0;jenkins-hbase4:39911] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 00:53:58,735 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 00:53:58,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249635818] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249635818,5,FailOnTimeoutGroup] 2023-02-13 00:53:58,735 DEBUG [M:0;jenkins-hbase4:39911] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 00:53:58,735 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249635819] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249635819,5,FailOnTimeoutGroup] 2023-02-13 00:53:58,736 INFO [M:0;jenkins-hbase4:39911] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 00:53:58,737 INFO [M:0;jenkins-hbase4:39911] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 00:53:58,737 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:58,737 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:58,737 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:58,737 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:53:58,737 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:53:58,738 DEBUG [Listener at localhost/35605] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 00:53:58,738 INFO [Listener at localhost/35605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38485,1676249635575' ***** 2023-02-13 00:53:58,738 INFO [Listener at localhost/35605] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 00:53:58,737 INFO [M:0;jenkins-hbase4:39911] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 00:53:58,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:58,738 INFO [Listener at localhost/35605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42839,1676249635601' ***** 2023-02-13 00:53:58,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:58,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:58,738 DEBUG [M:0;jenkins-hbase4:39911] master.HMaster(1502): Stopping service threads 2023-02-13 00:53:58,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:53:58,739 INFO [M:0;jenkins-hbase4:39911] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 00:53:58,738 INFO [RS:0;jenkins-hbase4:38485] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:53:58,739 INFO [M:0;jenkins-hbase4:39911] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 00:53:58,739 INFO [RS:0;jenkins-hbase4:38485] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:53:58,738 INFO [Listener at localhost/35605] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 00:53:58,740 ERROR [M:0;jenkins-hbase4:39911] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-13 00:53:58,740 INFO [RS:0;jenkins-hbase4:38485] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:53:58,740 INFO [M:0;jenkins-hbase4:39911] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 00:53:58,739 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:53:58,741 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 00:53:58,740 INFO [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(3304): Received CLOSE for 0a98b367f319e93352882eb969190eed 2023-02-13 00:53:58,740 INFO [RS:1;jenkins-hbase4:42839] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:53:58,740 INFO [Listener at localhost/35605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37023,1676249635625' ***** 2023-02-13 00:53:58,741 INFO [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:58,741 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:53:58,741 DEBUG [RS:0;jenkins-hbase4:38485] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53616874 to 127.0.0.1:57021 2023-02-13 00:53:58,741 INFO [Listener at localhost/35605] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 00:53:58,741 DEBUG [M:0;jenkins-hbase4:39911] zookeeper.ZKUtil(398): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 00:53:58,741 INFO [RS:1;jenkins-hbase4:42839] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:53:58,743 INFO [RS:2;jenkins-hbase4:37023] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:53:58,743 WARN [M:0;jenkins-hbase4:39911] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 00:53:58,743 INFO [RS:2;jenkins-hbase4:37023] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:53:58,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0a98b367f319e93352882eb969190eed, disabling compactions & flushes 2023-02-13 00:53:58,742 DEBUG [RS:0;jenkins-hbase4:38485] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:58,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:58,744 INFO [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 00:53:58,743 INFO [RS:2;jenkins-hbase4:37023] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:53:58,743 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:53:58,743 INFO [M:0;jenkins-hbase4:39911] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 00:53:58,743 INFO [RS:1;jenkins-hbase4:42839] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:53:58,744 INFO [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:58,744 DEBUG [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1479): Online Regions={0a98b367f319e93352882eb969190eed=restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.} 2023-02-13 00:53:58,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:58,745 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:58,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. after waiting 0 ms 2023-02-13 00:53:58,745 DEBUG [RS:2;jenkins-hbase4:37023] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75ae988a to 127.0.0.1:57021 2023-02-13 00:53:58,745 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(3304): Received CLOSE for 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:53:58,745 INFO [M:0;jenkins-hbase4:39911] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 00:53:58,746 DEBUG [RS:2;jenkins-hbase4:37023] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:58,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:58,747 DEBUG [M:0;jenkins-hbase4:39911] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 00:53:58,745 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:58,745 DEBUG [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1505): Waiting on 0a98b367f319e93352882eb969190eed 2023-02-13 00:53:58,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4534c163be465f052ed9a3af03edbb95, disabling compactions & flushes 2023-02-13 00:53:58,747 INFO [M:0;jenkins-hbase4:39911] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:58,746 INFO [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37023,1676249635625; all regions closed. 2023-02-13 00:53:58,746 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(3304): Received CLOSE for e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:58,748 DEBUG [M:0;jenkins-hbase4:39911] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:58,748 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(3304): Received CLOSE for e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:53:58,748 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:58,748 DEBUG [RS:1;jenkins-hbase4:42839] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66110ead to 127.0.0.1:57021 2023-02-13 00:53:58,748 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. after waiting 0 ms 2023-02-13 00:53:58,748 DEBUG [RS:1;jenkins-hbase4:42839] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:58,748 DEBUG [M:0;jenkins-hbase4:39911] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 00:53:58,748 DEBUG [M:0;jenkins-hbase4:39911] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:58,748 INFO [RS:1;jenkins-hbase4:42839] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:53:58,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,748 INFO [RS:1;jenkins-hbase4:42839] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:53:58,748 INFO [M:0;jenkins-hbase4:39911] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.18 KB heapSize=76.14 KB 2023-02-13 00:53:58,749 INFO [RS:1;jenkins-hbase4:42839] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:53:58,749 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 00:53:58,750 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1475): Waiting on 4 regions to close 2023-02-13 00:53:58,750 DEBUG [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1479): Online Regions={4534c163be465f052ed9a3af03edbb95=restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95., e95173dbc5ee3e5a46e6e44669d98b1f=hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f., e0dbd631ed14dc0143960d5a5baf5b84=restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 00:53:58,750 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 00:53:58,750 DEBUG [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1505): Waiting on 1588230740, 4534c163be465f052ed9a3af03edbb95, e0dbd631ed14dc0143960d5a5baf5b84, e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:53:58,750 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 00:53:58,750 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 00:53:58,751 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 00:53:58,751 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 00:53:58,751 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=5.14 KB heapSize=9.42 KB 2023-02-13 00:53:58,765 DEBUG [RS:2;jenkins-hbase4:37023] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs 2023-02-13 00:53:58,765 INFO [RS:2;jenkins-hbase4:37023] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37023%2C1676249635625:(num 1676249636116) 2023-02-13 00:53:58,765 DEBUG [RS:2;jenkins-hbase4:37023] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:58,765 INFO [RS:2;jenkins-hbase4:37023] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:58,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 00:53:58,768 INFO [RS:2;jenkins-hbase4:37023] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 00:53:58,768 INFO [RS:2;jenkins-hbase4:37023] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:53:58,768 INFO [RS:2;jenkins-hbase4:37023] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:53:58,768 INFO [RS:2;jenkins-hbase4:37023] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:53:58,768 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:58,769 INFO [RS:2;jenkins-hbase4:37023] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37023 2023-02-13 00:53:58,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:58,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0a98b367f319e93352882eb969190eed: Waiting for close lock at 1676249638743Running coprocessor pre-close hooks at 1676249638743Disabling compacts and flushes for region at 1676249638743Disabling writes for close at 1676249638746 (+3 ms)Writing region close event to WAL at 1676249638747 (+1 ms)Running coprocessor post-close hooks at 1676249638769 (+22 ms)Closed at 1676249638769 2023-02-13 00:53:58,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:53:58,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 00:53:58,772 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4534c163be465f052ed9a3af03edbb95: Waiting for close lock at 1676249638747Running coprocessor pre-close hooks at 1676249638747Disabling compacts and flushes for region at 1676249638747Disabling writes for close at 1676249638748 (+1 ms)Writing region close event to WAL at 1676249638749 (+1 ms)Running coprocessor post-close hooks at 1676249638772 (+23 ms)Closed at 1676249638772 2023-02-13 00:53:58,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:53:58,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e95173dbc5ee3e5a46e6e44669d98b1f, disabling compactions & flushes 2023-02-13 00:53:58,772 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:58,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:58,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. after waiting 0 ms 2023-02-13 00:53:58,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:58,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e95173dbc5ee3e5a46e6e44669d98b1f 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 00:53:58,784 INFO [M:0;jenkins-hbase4:39911] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.18 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/4e87b5086cd84c719b07502be9308a0e 2023-02-13 00:53:58,792 DEBUG [M:0;jenkins-hbase4:39911] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/4e87b5086cd84c719b07502be9308a0e as hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/4e87b5086cd84c719b07502be9308a0e 2023-02-13 00:53:58,796 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.77 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/.tmp/info/6b9b3e3781f24a4ab5b92c0459dba265 2023-02-13 00:53:58,803 INFO [M:0;jenkins-hbase4:39911] regionserver.HStore(1080): Added hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/4e87b5086cd84c719b07502be9308a0e, entries=17, sequenceid=141, filesize=8.5 K 2023-02-13 00:53:58,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/.tmp/info/f67aa219ce294fc9afb9b66e02ed8d3d 2023-02-13 00:53:58,804 INFO [M:0;jenkins-hbase4:39911] regionserver.HRegion(2947): Finished flush of dataSize ~64.18 KB/65719, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 56ms, sequenceid=141, compaction requested=false 2023-02-13 00:53:58,806 INFO [M:0;jenkins-hbase4:39911] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:53:58,806 DEBUG [M:0;jenkins-hbase4:39911] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676249638746Disabling compacts and flushes for region at 1676249638746Disabling writes for close at 1676249638748 (+2 ms)Obtaining lock to block concurrent updates at 1676249638749 (+1 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676249638749Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65719, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676249638749Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676249638751 (+2 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676249638752 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676249638762 (+10 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676249638762Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676249638793 (+31 ms)Finished flush of dataSize ~64.18 KB/65719, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 56ms, sequenceid=141, compaction requested=false at 1676249638804 (+11 ms)Writing region close event to WAL at 1676249638806 (+2 ms)Closed at 1676249638806 2023-02-13 00:53:58,812 INFO [M:0;jenkins-hbase4:39911] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 00:53:58,812 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:58,812 INFO [M:0;jenkins-hbase4:39911] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39911 2023-02-13 00:53:58,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/.tmp/info/f67aa219ce294fc9afb9b66e02ed8d3d as hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/info/f67aa219ce294fc9afb9b66e02ed8d3d 2023-02-13 00:53:58,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/info/f67aa219ce294fc9afb9b66e02ed8d3d, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 00:53:58,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e95173dbc5ee3e5a46e6e44669d98b1f in 55ms, sequenceid=6, compaction requested=false 2023-02-13 00:53:58,830 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:58,838 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/.tmp/table/69375cfd9b1e4ce99a8f74e7594aa4a6 2023-02-13 00:53:58,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 00:53:58,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:58,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e95173dbc5ee3e5a46e6e44669d98b1f: Waiting for close lock at 1676249638772Running coprocessor pre-close hooks at 1676249638772Disabling compacts and flushes for region at 1676249638772Disabling writes for close at 1676249638773 (+1 ms)Obtaining lock to block concurrent updates at 1676249638773Preparing flush snapshotting stores in e95173dbc5ee3e5a46e6e44669d98b1f at 1676249638773Finished memstore snapshotting hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676249638773Flushing stores of hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. at 1676249638774 (+1 ms)Flushing e95173dbc5ee3e5a46e6e44669d98b1f/info: creating writer at 1676249638774Flushing e95173dbc5ee3e5a46e6e44669d98b1f/info: appending metadata at 1676249638784 (+10 ms)Flushing e95173dbc5ee3e5a46e6e44669d98b1f/info: closing flushed file at 1676249638784Flushing e95173dbc5ee3e5a46e6e44669d98b1f/info: reopening flushed file at 1676249638822 (+38 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e95173dbc5ee3e5a46e6e44669d98b1f in 55ms, sequenceid=6, compaction requested=false at 1676249638829 (+7 ms)Writing region close event to WAL at 1676249638835 (+6 ms)Running coprocessor post-close hooks at 1676249638840 (+5 ms)Closed at 1676249638840 2023-02-13 00:53:58,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:53:58,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e0dbd631ed14dc0143960d5a5baf5b84, disabling compactions & flushes 2023-02-13 00:53:58,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:58,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:58,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. after waiting 0 ms 2023-02-13 00:53:58,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:58,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 00:53:58,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:58,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e0dbd631ed14dc0143960d5a5baf5b84: Waiting for close lock at 1676249638841Running coprocessor pre-close hooks at 1676249638841Disabling compacts and flushes for region at 1676249638841Disabling writes for close at 1676249638841Writing region close event to WAL at 1676249638841Running coprocessor post-close hooks at 1676249638846 (+5 ms)Closed at 1676249638846 2023-02-13 00:53:58,846 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/.tmp/info/6b9b3e3781f24a4ab5b92c0459dba265 as hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info/6b9b3e3781f24a4ab5b92c0459dba265 2023-02-13 00:53:58,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:53:58,851 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:58,851 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:58,851 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249635625 2023-02-13 00:53:58,851 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b2ad49 rejected from java.util.concurrent.ThreadPoolExecutor@78d29e7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:58,852 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:58,852 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:58,851 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:58,851 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:53:58,851 DEBUG [M:0;jenkins-hbase4:39911] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39911,1676249635523 already deleted, retry=false 2023-02-13 00:53:58,852 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@65c9344e rejected from java.util.concurrent.ThreadPoolExecutor@78d29e7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:58,855 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info/6b9b3e3781f24a4ab5b92c0459dba265, entries=40, sequenceid=24, filesize=9.5 K 2023-02-13 00:53:58,856 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/.tmp/table/69375cfd9b1e4ce99a8f74e7594aa4a6 as hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/table/69375cfd9b1e4ce99a8f74e7594aa4a6 2023-02-13 00:53:58,862 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/table/69375cfd9b1e4ce99a8f74e7594aa4a6, entries=8, sequenceid=24, filesize=5.0 K 2023-02-13 00:53:58,863 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 112ms, sequenceid=24, compaction requested=false 2023-02-13 00:53:58,869 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=1 2023-02-13 00:53:58,869 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 00:53:58,870 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 00:53:58,870 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676249638750Running coprocessor pre-close hooks at 1676249638750Disabling compacts and flushes for region at 1676249638750Disabling writes for close at 1676249638751 (+1 ms)Obtaining lock to block concurrent updates at 1676249638751Preparing flush snapshotting stores in 1588230740 at 1676249638751Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=5262, getHeapSize=9600, getOffHeapSize=0, getCellsCount=48 at 1676249638751Flushing stores of hbase:meta,,1.1588230740 at 1676249638759 (+8 ms)Flushing 1588230740/info: creating writer at 1676249638759Flushing 1588230740/info: appending metadata at 1676249638772 (+13 ms)Flushing 1588230740/info: closing flushed file at 1676249638772Flushing 1588230740/table: creating writer at 1676249638810 (+38 ms)Flushing 1588230740/table: appending metadata at 1676249638818 (+8 ms)Flushing 1588230740/table: closing flushed file at 1676249638818Flushing 1588230740/info: reopening flushed file at 1676249638847 (+29 ms)Flushing 1588230740/table: reopening flushed file at 1676249638857 (+10 ms)Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 112ms, sequenceid=24, compaction requested=false at 1676249638863 (+6 ms)Writing region close event to WAL at 1676249638866 (+3 ms)Running coprocessor post-close hooks at 1676249638869 (+3 ms)Closed at 1676249638870 (+1 ms) 2023-02-13 00:53:58,870 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 00:53:58,927 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 00:53:58,927 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 00:53:58,933 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 00:53:58,933 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 00:53:58,947 INFO [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38485,1676249635575; all regions closed. 2023-02-13 00:53:58,951 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42839,1676249635601; all regions closed. 2023-02-13 00:53:58,959 DEBUG [RS:0;jenkins-hbase4:38485] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs 2023-02-13 00:53:58,959 INFO [RS:0;jenkins-hbase4:38485] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38485%2C1676249635575:(num 1676249636118) 2023-02-13 00:53:58,959 DEBUG [RS:0;jenkins-hbase4:38485] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:58,960 INFO [RS:0;jenkins-hbase4:38485] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:58,960 INFO [RS:0;jenkins-hbase4:38485] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 00:53:58,960 INFO [RS:0;jenkins-hbase4:38485] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:53:58,960 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:58,960 INFO [RS:0;jenkins-hbase4:38485] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:53:58,960 INFO [RS:0;jenkins-hbase4:38485] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:53:58,961 INFO [RS:0;jenkins-hbase4:38485] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38485 2023-02-13 00:53:58,966 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:58,966 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:53:58,966 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1b6ac33b rejected from java.util.concurrent.ThreadPoolExecutor@1f55448e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:58,968 DEBUG [RS:1;jenkins-hbase4:42839] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs 2023-02-13 00:53:58,968 INFO [RS:1;jenkins-hbase4:42839] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42839%2C1676249635601.meta:.meta(num 1676249636228) 2023-02-13 00:53:58,974 DEBUG [RS:1;jenkins-hbase4:42839] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs 2023-02-13 00:53:58,975 INFO [RS:1;jenkins-hbase4:42839] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42839%2C1676249635601:(num 1676249636124) 2023-02-13 00:53:58,975 DEBUG [RS:1;jenkins-hbase4:42839] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:53:58,975 INFO [RS:1;jenkins-hbase4:42839] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:53:58,975 INFO [RS:1;jenkins-hbase4:42839] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 00:53:58,975 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:53:58,975 INFO [RS:1;jenkins-hbase4:42839] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42839 2023-02-13 00:53:58,983 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:53:58,983 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1b5e3347 rejected from java.util.concurrent.ThreadPoolExecutor@1f61a4f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:59,140 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:59,140 INFO [RS:1;jenkins-hbase4:42839] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42839,1676249635601; zookeeper connection closed. 2023-02-13 00:53:59,141 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@15d285b rejected from java.util.concurrent.ThreadPoolExecutor@1f61a4f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:59,141 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6d787b6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6d787b6 2023-02-13 00:53:59,141 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b0002, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:59,141 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7927a8b2 rejected from java.util.concurrent.ThreadPoolExecutor@1f61a4f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:59,240 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:59,240 INFO [RS:0;jenkins-hbase4:38485] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38485,1676249635575; zookeeper connection closed. 2023-02-13 00:53:59,241 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2679cf6b rejected from java.util.concurrent.ThreadPoolExecutor@1f55448e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:59,241 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@417c35b3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@417c35b3 2023-02-13 00:53:59,241 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b0001, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:59,241 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5e83eb56 rejected from java.util.concurrent.ThreadPoolExecutor@1f55448e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:59,341 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:59,341 INFO [M:0;jenkins-hbase4:39911] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39911,1676249635523; zookeeper connection closed. 2023-02-13 00:53:59,341 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5dd99a12 rejected from java.util.concurrent.ThreadPoolExecutor@59e60571[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:59,342 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:39911-0x1004b2b846b0000, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:59,342 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3c2a8bd2 rejected from java.util.concurrent.ThreadPoolExecutor@59e60571[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:59,441 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:59,441 INFO [RS:2;jenkins-hbase4:37023] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37023,1676249635625; zookeeper connection closed. 2023-02-13 00:53:59,441 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d2bc8dd rejected from java.util.concurrent.ThreadPoolExecutor@78d29e7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:59,441 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b0003, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:53:59,441 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@ee76363 rejected from java.util.concurrent.ThreadPoolExecutor@78d29e7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:53:59,441 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@66d51e8b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@66d51e8b 2023-02-13 00:53:59,442 INFO [Listener at localhost/35605] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-13 00:53:59,442 INFO [Listener at localhost/35605] master.TestRetainAssignmentOnRestart(93): Sleeping a bit 2023-02-13 00:54:01,442 INFO [Listener at localhost/35605] master.TestRetainAssignmentOnRestart(96): Starting cluster the second time with the same ports 2023-02-13 00:54:01,455 INFO [Listener at localhost/35605] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 00:54:01,455 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,455 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,455 INFO [Listener at localhost/35605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:54:01,455 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,455 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:54:01,455 INFO [Listener at localhost/35605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:54:01,456 INFO [Listener at localhost/35605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35729 2023-02-13 00:54:01,457 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:54:01,458 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:54:01,459 INFO [Listener at localhost/35605] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35729 connecting to ZooKeeper ensemble=127.0.0.1:57021 2023-02-13 00:54:01,463 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:357290x0, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:54:01,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:35729-0x1004b2b846b0009 connected 2023-02-13 00:54:01,466 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:54:01,467 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(162): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 00:54:01,467 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:54:01,468 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35729 2023-02-13 00:54:01,468 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35729 2023-02-13 00:54:01,468 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35729 2023-02-13 00:54:01,469 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35729 2023-02-13 00:54:01,469 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35729 2023-02-13 00:54:01,469 INFO [Listener at localhost/35605] master.HMaster(439): hbase.rootdir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc, hbase.cluster.distributed=false 2023-02-13 00:54:01,469 DEBUG [pool-506-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-13 00:54:01,471 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,472 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 00:54:01,473 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,474 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 00:54:01,474 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:54:01,475 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:54:01,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:54:01,475 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,35729,1676249641454 from backup master directory 2023-02-13 00:54:01,475 INFO [M:1;jenkins-hbase4:35729] regionserver.HRegionServer(952): ClusterId : dfbf218d-5af2-44f7-8fcf-91e5be54a5ab 2023-02-13 00:54:01,477 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,477 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 00:54:01,477 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:54:01,477 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,486 INFO [Listener at localhost/35605] client.ConnectionUtils(127): regionserver/jenkins-hbase4:38485 server-side Connection retries=45 2023-02-13 00:54:01,486 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,486 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,486 INFO [Listener at localhost/35605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:54:01,486 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,486 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:54:01,487 INFO [Listener at localhost/35605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:54:01,487 INFO [Listener at localhost/35605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38485 2023-02-13 00:54:01,488 INFO [Listener at localhost/35605] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:54:01,494 DEBUG [Listener at localhost/35605] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:54:01,495 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:54:01,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:54:01,496 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:54:01,497 INFO [Listener at localhost/35605] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38485 connecting to ZooKeeper ensemble=127.0.0.1:57021 2023-02-13 00:54:01,500 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:384850x0, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:54:01,501 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(162): regionserver:384850x0, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:54:01,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38485-0x1004b2b846b000a connected 2023-02-13 00:54:01,502 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 00:54:01,503 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:54:01,504 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38485 2023-02-13 00:54:01,504 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38485 2023-02-13 00:54:01,505 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38485 2023-02-13 00:54:01,505 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38485 2023-02-13 00:54:01,506 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38485 2023-02-13 00:54:01,507 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(952): ClusterId : dfbf218d-5af2-44f7-8fcf-91e5be54a5ab 2023-02-13 00:54:01,507 DEBUG [RS:3;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:54:01,509 DEBUG [RS:3;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:54:01,509 DEBUG [RS:3;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:54:01,516 DEBUG [RS:3;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:54:01,517 DEBUG [RS:3;jenkins-hbase4:38485] zookeeper.ReadOnlyZKClient(139): Connect 0x6424be94 to 127.0.0.1:57021 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:54:01,521 DEBUG [RS:3;jenkins-hbase4:38485] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ae34c60, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:54:01,521 DEBUG [RS:3;jenkins-hbase4:38485] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edcd5e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:54:01,529 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6465eb61 to 127.0.0.1:57021 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:54:01,532 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d7f88, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:54:01,532 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase4:38485 2023-02-13 00:54:01,532 INFO [RS:3;jenkins-hbase4:38485] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:54:01,532 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 00:54:01,532 INFO [RS:3;jenkins-hbase4:38485] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:54:01,532 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:54:01,533 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35729,1676249641454 with isa=jenkins-hbase4.apache.org/172.31.14.131:38485, startcode=1676249641485 2023-02-13 00:54:01,533 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 00:54:01,533 DEBUG [RS:3;jenkins-hbase4:38485] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:54:01,533 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:54:01,536 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44181, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.9 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:54:01,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:54:01,538 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 00:54:01,538 WARN [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 00:54:01,540 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,39911,1676249635523 to hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,39911,1676249635523-dead as it is dead 2023-02-13 00:54:01,541 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,39911,1676249635523-dead/jenkins-hbase4.apache.org%2C39911%2C1676249635523.1676249635727 2023-02-13 00:54:01,541 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,39911,1676249635523-dead/jenkins-hbase4.apache.org%2C39911%2C1676249635523.1676249635727 after 0ms 2023-02-13 00:54:01,542 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,39911,1676249635523-dead/jenkins-hbase4.apache.org%2C39911%2C1676249635523.1676249635727 to hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39911%2C1676249635523.1676249635727 2023-02-13 00:54:01,542 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,39911,1676249635523-dead 2023-02-13 00:54:01,542 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,544 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35729%2C1676249641454, suffix=, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,35729,1676249641454, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/oldWALs, maxLogs=10 2023-02-13 00:54:01,557 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:54:01,561 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:54:01,561 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:54:01,563 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/WALs/jenkins-hbase4.apache.org,35729,1676249641454/jenkins-hbase4.apache.org%2C35729%2C1676249641454.1676249641545 2023-02-13 00:54:01,564 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK], DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK], DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK]] 2023-02-13 00:54:01,564 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:54:01,564 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:54:01,564 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:54:01,564 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:54:01,565 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:54:01,566 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 00:54:01,567 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 00:54:01,576 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/4e87b5086cd84c719b07502be9308a0e 2023-02-13 00:54:01,576 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:54:01,577 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-13 00:54:01,577 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39911%2C1676249635523.1676249635727 2023-02-13 00:54:01,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39911%2C1676249635523.1676249635727 2023-02-13 00:54:01,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39911%2C1676249635523.1676249635727 2023-02-13 00:54:01,587 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 00:54:01,590 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-13 00:54:01,591 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62828537, jitterRate=-0.06378184258937836}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:54:01,591 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676249641564Initializing all the Stores at 1676249641565 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249641565Cleaning up temporary data from old regions at 1676249641585 (+20 ms)Cleaning up detritus from prior splits at 1676249641585Region opened successfully at 1676249641591 (+6 ms) 2023-02-13 00:54:01,591 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 00:54:01,592 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 00:54:01,592 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 00:54:01,592 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 00:54:01,593 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 00:54:01,597 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-13 00:54:01,597 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-13 00:54:01,597 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-13 00:54:01,597 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-13 00:54:01,598 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-13 00:54:01,598 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-13 00:54:01,598 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-13 00:54:01,598 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-13 00:54:01,598 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 00:54:01,599 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-13 00:54:01,600 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,42839,1676249635601, table=hbase:meta, region=1588230740 2023-02-13 00:54:01,601 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-13 00:54:01,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37023,1676249635625 already deleted, retry=false 2023-02-13 00:54:01,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,37023,1676249635625 on jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,604 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,37023,1676249635625, splitWal=true, meta=false 2023-02-13 00:54:01,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase4.apache.org,37023,1676249635625 (carryingMeta=false) jenkins-hbase4.apache.org,37023,1676249635625/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1011676a[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 00:54:01,606 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38485,1676249635575 already deleted, retry=false 2023-02-13 00:54:01,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,38485,1676249635575 on jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,607 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,38485,1676249635575, splitWal=true, meta=false 2023-02-13 00:54:01,607 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=19 for jenkins-hbase4.apache.org,38485,1676249635575 (carryingMeta=false) jenkins-hbase4.apache.org,38485,1676249635575/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1f2813e7[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 00:54:01,608 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42839,1676249635601 already deleted, retry=false 2023-02-13 00:54:01,608 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,42839,1676249635601 on jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,609 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=20, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,42839,1676249635601, splitWal=true, meta=true 2023-02-13 00:54:01,609 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=20 for jenkins-hbase4.apache.org,42839,1676249635601 (carryingMeta=true) jenkins-hbase4.apache.org,42839,1676249635601/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1b3de093[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 00:54:01,619 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 00:54:01,619 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 00:54:01,620 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-13 00:54:01,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 00:54:01,621 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 00:54:01,621 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 00:54:01,622 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 00:54:01,622 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 00:54:01,622 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,35729,1676249641454, sessionid=0x1004b2b846b0009, setting cluster-up flag (Was=true) 2023-02-13 00:54:01,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 00:54:01,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 00:54:01,628 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:01,629 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/.hbase-snapshot/.tmp 2023-02-13 00:54:01,629 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:54:01,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:54:01,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:54:01,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 00:54:01,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 00:54:01,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 00:54:01,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,639 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35729,1676249641454 with isa=jenkins-hbase4.apache.org/172.31.14.131:38485, startcode=1676249641485 2023-02-13 00:54:01,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:54:01,639 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 00:54:01,640 WARN [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 200 ms and then retrying. 2023-02-13 00:54:01,641 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676249671641 2023-02-13 00:54:01,641 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 00:54:01,642 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42839,1676249635601; numProcessing=1 2023-02-13 00:54:01,642 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 00:54:01,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 00:54:01,643 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=20, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42839,1676249635601, splitWal=true, meta=true 2023-02-13 00:54:01,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 00:54:01,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 00:54:01,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 00:54:01,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,645 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 00:54:01,645 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 00:54:01,645 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 00:54:01,645 DEBUG [PEWorker-2] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37023,1676249635625; numProcessing=2 2023-02-13 00:54:01,645 DEBUG [PEWorker-3] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38485,1676249635575; numProcessing=3 2023-02-13 00:54:01,645 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,37023,1676249635625, splitWal=true, meta=false 2023-02-13 00:54:01,645 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=20, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42839,1676249635601, splitWal=true, meta=true, isMeta: true 2023-02-13 00:54:01,645 INFO [PEWorker-3] procedure.ServerCrashProcedure(161): Start pid=19, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,38485,1676249635575, splitWal=true, meta=false 2023-02-13 00:54:01,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 00:54:01,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 00:54:01,654 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601-splitting 2023-02-13 00:54:01,655 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601-splitting dir is empty, no logs to split. 2023-02-13 00:54:01,655 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,42839,1676249635601 WAL count=0, meta=true 2023-02-13 00:54:01,657 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601-splitting dir is empty, no logs to split. 2023-02-13 00:54:01,657 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,42839,1676249635601 WAL count=0, meta=true 2023-02-13 00:54:01,657 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42839,1676249635601 WAL splitting is done? wals=0, meta=true 2023-02-13 00:54:01,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 00:54:01,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249641653,5,FailOnTimeoutGroup] 2023-02-13 00:54:01,662 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249641660,5,FailOnTimeoutGroup] 2023-02-13 00:54:01,663 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 00:54:01,663 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,663 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 00:54:01,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676249641473, completionTime=-1 2023-02-13 00:54:01,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=0; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 00:54:01,670 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,42839,1676249635601; forceNewPlan=false, retain=false 2023-02-13 00:54:01,748 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-13 00:54:01,821 WARN [jenkins-hbase4:35729] assignment.AssignmentManager(2157): No servers available; cannot place 1 unassigned regions. 2023-02-13 00:54:01,840 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35729,1676249641454 with isa=jenkins-hbase4.apache.org/172.31.14.131:38485, startcode=1676249641485 2023-02-13 00:54:01,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:01,842 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc 2023-02-13 00:54:01,842 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44257 2023-02-13 00:54:01,842 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:54:01,844 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:01,845 DEBUG [RS:3;jenkins-hbase4:38485] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:01,845 WARN [RS:3;jenkins-hbase4:38485] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:54:01,845 INFO [RS:3;jenkins-hbase4:38485] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:54:01,845 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:01,846 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38485,1676249641485] 2023-02-13 00:54:01,850 DEBUG [RS:3;jenkins-hbase4:38485] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:01,850 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:54:01,850 INFO [RS:3;jenkins-hbase4:38485] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:54:01,852 INFO [RS:3;jenkins-hbase4:38485] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:54:01,852 INFO [RS:3;jenkins-hbase4:38485] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:54:01,852 INFO [RS:3;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,852 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:54:01,853 INFO [RS:3;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:38485, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:38485, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:38485, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:38485, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:38485, corePoolSize=2, maxPoolSize=2 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:38485, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:38485, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:38485, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,854 DEBUG [RS:3;jenkins-hbase4:38485] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:38485, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,855 INFO [RS:3;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,855 INFO [RS:3;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,855 INFO [RS:3;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,864 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=1; waited=200ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 00:54:01,866 INFO [RS:3;jenkins-hbase4:38485] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:54:01,866 INFO [RS:3;jenkins-hbase4:38485] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38485,1676249641485-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,876 INFO [RS:3;jenkins-hbase4:38485] regionserver.Replication(203): jenkins-hbase4.apache.org,38485,1676249641485 started 2023-02-13 00:54:01,876 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38485,1676249641485, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38485, sessionid=0x1004b2b846b000a 2023-02-13 00:54:01,876 DEBUG [RS:3;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:54:01,876 DEBUG [RS:3;jenkins-hbase4:38485] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:01,876 DEBUG [RS:3;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38485,1676249641485' 2023-02-13 00:54:01,876 DEBUG [RS:3;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:54:01,877 DEBUG [RS:3;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:54:01,877 DEBUG [RS:3;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:54:01,877 DEBUG [RS:3;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:54:01,877 DEBUG [RS:3;jenkins-hbase4:38485] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:01,877 DEBUG [RS:3;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38485,1676249641485' 2023-02-13 00:54:01,877 DEBUG [RS:3;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:54:01,877 DEBUG [RS:3;jenkins-hbase4:38485] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:54:01,878 DEBUG [RS:3;jenkins-hbase4:38485] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:54:01,878 INFO [RS:3;jenkins-hbase4:38485] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:54:01,878 INFO [RS:3;jenkins-hbase4:38485] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:54:01,886 INFO [Listener at localhost/35605] client.ConnectionUtils(127): regionserver/jenkins-hbase4:42839 server-side Connection retries=45 2023-02-13 00:54:01,886 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,887 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,887 INFO [Listener at localhost/35605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:54:01,887 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,887 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:54:01,887 INFO [Listener at localhost/35605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:54:01,887 INFO [Listener at localhost/35605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42839 2023-02-13 00:54:01,888 INFO [Listener at localhost/35605] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:54:01,889 DEBUG [Listener at localhost/35605] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:54:01,889 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:54:01,890 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:54:01,891 INFO [Listener at localhost/35605] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42839 connecting to ZooKeeper ensemble=127.0.0.1:57021 2023-02-13 00:54:01,895 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:428390x0, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:54:01,896 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(162): regionserver:428390x0, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:54:01,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42839-0x1004b2b846b000d connected 2023-02-13 00:54:01,897 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 00:54:01,898 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:54:01,898 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42839 2023-02-13 00:54:01,900 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42839 2023-02-13 00:54:01,904 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42839 2023-02-13 00:54:01,904 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42839 2023-02-13 00:54:01,904 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42839 2023-02-13 00:54:01,906 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(952): ClusterId : dfbf218d-5af2-44f7-8fcf-91e5be54a5ab 2023-02-13 00:54:01,906 DEBUG [RS:4;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:54:01,908 DEBUG [RS:4;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:54:01,908 DEBUG [RS:4;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:54:01,911 DEBUG [RS:4;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:54:01,912 DEBUG [RS:4;jenkins-hbase4:42839] zookeeper.ReadOnlyZKClient(139): Connect 0x265aa063 to 127.0.0.1:57021 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:54:01,916 DEBUG [RS:4;jenkins-hbase4:42839] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@289d5ad8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:54:01,916 DEBUG [RS:4;jenkins-hbase4:42839] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ca2cb13, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:54:01,924 DEBUG [RS:4;jenkins-hbase4:42839] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:4;jenkins-hbase4:42839 2023-02-13 00:54:01,924 INFO [RS:4;jenkins-hbase4:42839] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:54:01,924 INFO [RS:4;jenkins-hbase4:42839] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:54:01,924 DEBUG [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:54:01,925 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35729,1676249641454 with isa=jenkins-hbase4.apache.org/172.31.14.131:42839, startcode=1676249641886 2023-02-13 00:54:01,925 DEBUG [RS:4;jenkins-hbase4:42839] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:54:01,927 INFO [RS-EventLoopGroup-13-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49399, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.10 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:54:01,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:01,928 DEBUG [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc 2023-02-13 00:54:01,928 DEBUG [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44257 2023-02-13 00:54:01,928 DEBUG [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:54:01,929 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:01,929 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:01,930 DEBUG [RS:4;jenkins-hbase4:42839] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:01,930 WARN [RS:4;jenkins-hbase4:42839] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:54:01,930 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42839,1676249641886] 2023-02-13 00:54:01,930 INFO [RS:4;jenkins-hbase4:42839] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:54:01,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:01,930 DEBUG [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:01,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:01,934 DEBUG [RS:4;jenkins-hbase4:42839] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:01,934 DEBUG [RS:4;jenkins-hbase4:42839] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:01,935 DEBUG [RS:4;jenkins-hbase4:42839] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:54:01,935 INFO [RS:4;jenkins-hbase4:42839] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:54:01,937 INFO [RS:4;jenkins-hbase4:42839] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:54:01,937 INFO [RS:4;jenkins-hbase4:42839] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:54:01,937 INFO [RS:4;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,937 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:54:01,938 INFO [RS:4;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,938 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:42839, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,938 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:42839, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,938 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,938 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,938 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:42839, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,938 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:42839, corePoolSize=2, maxPoolSize=2 2023-02-13 00:54:01,939 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:42839, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,939 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:42839, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,939 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:42839, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,939 DEBUG [RS:4;jenkins-hbase4:42839] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:42839, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:01,939 INFO [RS:4;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,939 INFO [RS:4;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,940 INFO [RS:4;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,950 INFO [RS:4;jenkins-hbase4:42839] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:54:01,950 INFO [RS:4;jenkins-hbase4:42839] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42839,1676249641886-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:01,961 INFO [RS:4;jenkins-hbase4:42839] regionserver.Replication(203): jenkins-hbase4.apache.org,42839,1676249641886 started 2023-02-13 00:54:01,961 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42839,1676249641886, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42839, sessionid=0x1004b2b846b000d 2023-02-13 00:54:01,961 DEBUG [RS:4;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:54:01,961 DEBUG [RS:4;jenkins-hbase4:42839] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:01,961 DEBUG [RS:4;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42839,1676249641886' 2023-02-13 00:54:01,961 DEBUG [RS:4;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:54:01,962 DEBUG [RS:4;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:54:01,962 DEBUG [RS:4;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:54:01,962 DEBUG [RS:4;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:54:01,962 DEBUG [RS:4;jenkins-hbase4:42839] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:01,962 DEBUG [RS:4;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42839,1676249641886' 2023-02-13 00:54:01,962 DEBUG [RS:4;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:54:01,962 DEBUG [RS:4;jenkins-hbase4:42839] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:54:01,963 DEBUG [RS:4;jenkins-hbase4:42839] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:54:01,963 INFO [RS:4;jenkins-hbase4:42839] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:54:01,963 INFO [RS:4;jenkins-hbase4:42839] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:54:01,964 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=300ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 00:54:01,972 INFO [Listener at localhost/35605] client.ConnectionUtils(127): regionserver/jenkins-hbase4:37023 server-side Connection retries=45 2023-02-13 00:54:01,972 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,972 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,972 INFO [Listener at localhost/35605] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 00:54:01,972 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 00:54:01,972 INFO [Listener at localhost/35605] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 00:54:01,972 INFO [Listener at localhost/35605] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 00:54:01,973 INFO [Listener at localhost/35605] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37023 2023-02-13 00:54:01,973 INFO [Listener at localhost/35605] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 00:54:01,975 DEBUG [Listener at localhost/35605] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 00:54:01,976 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:54:01,977 INFO [Listener at localhost/35605] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 00:54:01,977 INFO [Listener at localhost/35605] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37023 connecting to ZooKeeper ensemble=127.0.0.1:57021 2023-02-13 00:54:01,981 INFO [RS:3;jenkins-hbase4:38485] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38485%2C1676249641485, suffix=, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249641485, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs, maxLogs=32 2023-02-13 00:54:01,982 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:370230x0, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 00:54:01,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37023-0x1004b2b846b000f connected 2023-02-13 00:54:01,985 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 00:54:01,986 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 00:54:01,986 DEBUG [Listener at localhost/35605] zookeeper.ZKUtil(164): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 00:54:01,987 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37023 2023-02-13 00:54:01,988 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37023 2023-02-13 00:54:01,988 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37023 2023-02-13 00:54:01,988 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37023 2023-02-13 00:54:01,989 DEBUG [Listener at localhost/35605] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37023 2023-02-13 00:54:01,990 INFO [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(952): ClusterId : dfbf218d-5af2-44f7-8fcf-91e5be54a5ab 2023-02-13 00:54:01,990 DEBUG [RS:5;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 00:54:01,993 DEBUG [RS:5;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 00:54:01,993 DEBUG [RS:5;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 00:54:01,994 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 00:54:01,995 DEBUG [RS:5;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 00:54:01,997 DEBUG [RS:5;jenkins-hbase4:37023] zookeeper.ReadOnlyZKClient(139): Connect 0x65a5bbbe to 127.0.0.1:57021 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 00:54:02,000 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:54:02,001 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:54:02,001 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:54:02,004 DEBUG [RS:5;jenkins-hbase4:37023] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@357f971e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 00:54:02,004 DEBUG [RS:5;jenkins-hbase4:37023] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@548a1cad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:54:02,004 INFO [RS:3;jenkins-hbase4:38485] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249641485/jenkins-hbase4.apache.org%2C38485%2C1676249641485.1676249641981 2023-02-13 00:54:02,004 DEBUG [RS:3;jenkins-hbase4:38485] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK], DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK], DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK]] 2023-02-13 00:54:02,013 DEBUG [RS:5;jenkins-hbase4:37023] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:5;jenkins-hbase4:37023 2023-02-13 00:54:02,013 INFO [RS:5;jenkins-hbase4:37023] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 00:54:02,013 INFO [RS:5;jenkins-hbase4:37023] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 00:54:02,013 DEBUG [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 00:54:02,013 INFO [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35729,1676249641454 with isa=jenkins-hbase4.apache.org/172.31.14.131:37023, startcode=1676249641971 2023-02-13 00:54:02,014 DEBUG [RS:5;jenkins-hbase4:37023] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 00:54:02,015 INFO [RS-EventLoopGroup-13-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46593, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.11 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 00:54:02,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:02,016 DEBUG [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc 2023-02-13 00:54:02,016 DEBUG [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44257 2023-02-13 00:54:02,016 DEBUG [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 00:54:02,017 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:02,017 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:02,017 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:02,018 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37023,1676249641971] 2023-02-13 00:54:02,018 DEBUG [RS:5;jenkins-hbase4:37023] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:02,018 WARN [RS:5;jenkins-hbase4:37023] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 00:54:02,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:02,018 INFO [RS:5;jenkins-hbase4:37023] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:54:02,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:02,018 DEBUG [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:02,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:02,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:02,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:02,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:02,024 DEBUG [RS:5;jenkins-hbase4:37023] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:02,024 DEBUG [RS:5;jenkins-hbase4:37023] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:02,025 DEBUG [RS:5;jenkins-hbase4:37023] zookeeper.ZKUtil(162): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:02,025 DEBUG [RS:5;jenkins-hbase4:37023] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 00:54:02,025 INFO [RS:5;jenkins-hbase4:37023] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 00:54:02,027 INFO [RS:5;jenkins-hbase4:37023] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 00:54:02,027 INFO [RS:5;jenkins-hbase4:37023] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 00:54:02,027 INFO [RS:5;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:02,027 INFO [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 00:54:02,029 INFO [RS:5;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:02,029 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:37023, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:02,029 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:37023, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:02,029 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:37023, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:02,029 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:37023, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:02,029 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:37023, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:02,029 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:37023, corePoolSize=2, maxPoolSize=2 2023-02-13 00:54:02,029 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:37023, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:02,029 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:37023, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:02,029 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:37023, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:02,030 DEBUG [RS:5;jenkins-hbase4:37023] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:37023, corePoolSize=1, maxPoolSize=1 2023-02-13 00:54:02,030 INFO [RS:5;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:02,030 INFO [RS:5;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:02,030 INFO [RS:5;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:02,041 INFO [RS:5;jenkins-hbase4:37023] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 00:54:02,041 INFO [RS:5;jenkins-hbase4:37023] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37023,1676249641971-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:02,052 INFO [RS:5;jenkins-hbase4:37023] regionserver.Replication(203): jenkins-hbase4.apache.org,37023,1676249641971 started 2023-02-13 00:54:02,052 INFO [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,37023,1676249641971, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37023, sessionid=0x1004b2b846b000f 2023-02-13 00:54:02,052 DEBUG [RS:5;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 00:54:02,052 DEBUG [RS:5;jenkins-hbase4:37023] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:02,052 DEBUG [RS:5;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37023,1676249641971' 2023-02-13 00:54:02,052 DEBUG [RS:5;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 00:54:02,052 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:54:02,052 DEBUG [RS:5;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 00:54:02,053 DEBUG [RS:5;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 00:54:02,053 DEBUG [RS:5;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 00:54:02,053 DEBUG [RS:5;jenkins-hbase4:37023] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:02,053 DEBUG [RS:5;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37023,1676249641971' 2023-02-13 00:54:02,053 DEBUG [RS:5;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 00:54:02,053 DEBUG [RS:5;jenkins-hbase4:37023] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 00:54:02,054 DEBUG [RS:5;jenkins-hbase4:37023] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 00:54:02,054 INFO [RS:5;jenkins-hbase4:37023] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 00:54:02,054 INFO [RS:5;jenkins-hbase4:37023] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 00:54:02,054 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37048, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 00:54:02,055 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42839] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:37048 deadline: 1676249702054, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:02,064 INFO [RS:4;jenkins-hbase4:42839] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42839%2C1676249641886, suffix=, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249641886, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs, maxLogs=32 2023-02-13 00:54:02,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=401ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 00:54:02,065 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1676249641671, server=jenkins-hbase4.apache.org,42839,1676249635601}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 00:54:02,071 INFO [jenkins-hbase4:35729] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 00:54:02,072 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42839,1676249641886, state=OPENING 2023-02-13 00:54:02,074 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 00:54:02,074 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 00:54:02,075 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42839,1676249641886}] 2023-02-13 00:54:02,084 DEBUG [RS-EventLoopGroup-15-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:54:02,084 DEBUG [RS-EventLoopGroup-15-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:54:02,084 DEBUG [RS-EventLoopGroup-15-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:54:02,088 INFO [RS:4;jenkins-hbase4:42839] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249641886/jenkins-hbase4.apache.org%2C42839%2C1676249641886.1676249642065 2023-02-13 00:54:02,089 DEBUG [RS:4;jenkins-hbase4:42839] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK], DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK], DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK]] 2023-02-13 00:54:02,156 INFO [RS:5;jenkins-hbase4:37023] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37023%2C1676249641971, suffix=, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249641971, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs, maxLogs=32 2023-02-13 00:54:02,156 WARN [ReadOnlyZKClient-127.0.0.1:57021@0x7ea9bbe5] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-02-13 00:54:02,157 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42839] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:37048 deadline: 1676249702157, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:02,177 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:54:02,177 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:54:02,177 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:54:02,181 INFO [RS:5;jenkins-hbase4:37023] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249641971/jenkins-hbase4.apache.org%2C37023%2C1676249641971.1676249642157 2023-02-13 00:54:02,182 DEBUG [RS:5;jenkins-hbase4:37023] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK], DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK], DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK]] 2023-02-13 00:54:02,232 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:02,232 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 00:54:02,233 INFO [RS-EventLoopGroup-15-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37062, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-13 00:54:02,237 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 00:54:02,237 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 00:54:02,238 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42839%2C1676249641886.meta, suffix=.meta, logDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249641886, archiveDir=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs, maxLogs=32 2023-02-13 00:54:02,252 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK] 2023-02-13 00:54:02,252 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK] 2023-02-13 00:54:02,252 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK] 2023-02-13 00:54:02,254 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249641886/jenkins-hbase4.apache.org%2C42839%2C1676249641886.meta.1676249642239.meta 2023-02-13 00:54:02,254 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46265,DS-5beb4e02-174a-467b-aab6-05de5e0deb30,DISK], DatanodeInfoWithStorage[127.0.0.1:36655,DS-f5060a68-6f1a-4840-a502-240687ab4501,DISK], DatanodeInfoWithStorage[127.0.0.1:35619,DS-17f61d75-c836-4da5-b671-d9a9297ad34b,DISK]] 2023-02-13 00:54:02,254 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:54:02,254 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 00:54:02,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 00:54:02,255 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 00:54:02,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 00:54:02,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:54:02,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 00:54:02,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 00:54:02,260 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 00:54:02,261 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info 2023-02-13 00:54:02,261 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info 2023-02-13 00:54:02,261 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 00:54:02,267 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info/6b9b3e3781f24a4ab5b92c0459dba265 2023-02-13 00:54:02,267 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:54:02,267 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 00:54:02,268 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:54:02,268 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/rep_barrier 2023-02-13 00:54:02,269 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 00:54:02,269 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:54:02,269 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 00:54:02,270 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/table 2023-02-13 00:54:02,270 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/table 2023-02-13 00:54:02,270 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 00:54:02,276 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/table/69375cfd9b1e4ce99a8f74e7594aa4a6 2023-02-13 00:54:02,276 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:54:02,277 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740 2023-02-13 00:54:02,278 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740 2023-02-13 00:54:02,280 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 00:54:02,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 00:54:02,282 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=28; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68552218, jitterRate=0.02150765061378479}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 00:54:02,283 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676249642255Writing region info on filesystem at 1676249642255Initializing all the Stores at 1676249642256 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249642256Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249642260 (+4 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249642260Cleaning up temporary data from old regions at 1676249642279 (+19 ms)Cleaning up detritus from prior splits at 1676249642280 (+1 ms)Running coprocessor post-open hooks at 1676249642283 (+3 ms)Region opened successfully at 1676249642283 2023-02-13 00:54:02,283 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=22, masterSystemTime=1676249642232 2023-02-13 00:54:02,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 00:54:02,287 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:42839-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 00:54:02,288 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42839,1676249641886, state=OPEN 2023-02-13 00:54:02,289 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 00:54:02,290 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 00:54:02,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-02-13 00:54:02,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42839,1676249641886 in 216 msec 2023-02-13 00:54:02,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-02-13 00:54:02,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 633 msec 2023-02-13 00:54:02,370 WARN [RPCClient-NioEventLoopGroup-6-6] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39911 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39911 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:54:02,371 WARN [Listener at localhost/35605] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:39911 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39911 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39911 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 00:54:02,371 DEBUG [RPCClient-NioEventLoopGroup-6-6] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39911 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39911 2023-02-13 00:54:02,372 DEBUG [Listener at localhost/35605] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39911 this server is in the failed servers list 2023-02-13 00:54:02,372 WARN [Listener at localhost/35605] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:39911 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:39911 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnClusterRestart(TestRetainAssignmentOnRestart.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:39911 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-13 00:54:02,373 DEBUG [Listener at localhost/35605] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 00:54:02,374 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43550, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 00:54:02,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:54:02,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.CallRunner(144): callId: 50 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:43550 deadline: 1676249702375, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:54:02,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:54:02,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.CallRunner(144): callId: 52 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:43550 deadline: 1676249702477, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:54:02,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:54:02,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.CallRunner(144): callId: 54 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:43550 deadline: 1676249702679, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:54:02,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:54:02,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.CallRunner(144): callId: 56 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:43550 deadline: 1676249702982, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:54:03,065 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 00:54:03,069 DEBUG [hconnection-0x2fe60ec7-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 00:54:03,071 INFO [RS-EventLoopGroup-15-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37078, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=ClientService 2023-02-13 00:54:03,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=e95173dbc5ee3e5a46e6e44669d98b1f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,42839,1676249635601, regionLocation=jenkins-hbase4.apache.org,42839,1676249635601, openSeqNum=2 2023-02-13 00:54:03,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=e0dbd631ed14dc0143960d5a5baf5b84, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,42839,1676249635601, regionLocation=jenkins-hbase4.apache.org,42839,1676249635601, openSeqNum=2 2023-02-13 00:54:03,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=4534c163be465f052ed9a3af03edbb95, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,42839,1676249635601, regionLocation=jenkins-hbase4.apache.org,42839,1676249635601, openSeqNum=2 2023-02-13 00:54:03,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=0a98b367f319e93352882eb969190eed, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,38485,1676249635575, regionLocation=jenkins-hbase4.apache.org,38485,1676249635575, openSeqNum=2 2023-02-13 00:54:03,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 00:54:03,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676249703074 2023-02-13 00:54:03,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676249763074 2023-02-13 00:54:03,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 9 msec 2023-02-13 00:54:03,097 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,42839,1676249635601 had 4 regions 2023-02-13 00:54:03,098 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35729,1676249641454-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:03,098 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,38485,1676249635575 had 1 regions 2023-02-13 00:54:03,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35729,1676249641454-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:03,098 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,37023,1676249635625 had 0 regions 2023-02-13 00:54:03,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35729,1676249641454-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:03,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:35729, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:03,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 00:54:03,099 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. is NOT online; state={e95173dbc5ee3e5a46e6e44669d98b1f state=OPEN, ts=1676249643074, server=jenkins-hbase4.apache.org,42839,1676249635601}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 00:54:03,100 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,38485,1676249635575, splitWal=true, meta=false, isMeta: false 2023-02-13 00:54:03,101 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=20, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42839,1676249635601, splitWal=true, meta=true, isMeta: false 2023-02-13 00:54:03,101 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,37023,1676249635625, splitWal=true, meta=false, isMeta: false 2023-02-13 00:54:03,103 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249635575-splitting 2023-02-13 00:54:03,104 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249635575-splitting dir is empty, no logs to split. 2023-02-13 00:54:03,104 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,38485,1676249635575 WAL count=0, meta=false 2023-02-13 00:54:03,106 WARN [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase4.apache.org,42839,1676249635601/hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f., unknown_server=jenkins-hbase4.apache.org,42839,1676249635601/restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84., unknown_server=jenkins-hbase4.apache.org,42839,1676249635601/restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95., unknown_server=jenkins-hbase4.apache.org,38485,1676249635575/restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:54:03,106 DEBUG [PEWorker-3] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249635625-splitting 2023-02-13 00:54:03,107 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601-splitting dir is empty, no logs to split. 2023-02-13 00:54:03,107 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,42839,1676249635601 WAL count=0, meta=false 2023-02-13 00:54:03,109 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249635575-splitting dir is empty, no logs to split. 2023-02-13 00:54:03,109 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,38485,1676249635575 WAL count=0, meta=false 2023-02-13 00:54:03,109 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249635625-splitting dir is empty, no logs to split. 2023-02-13 00:54:03,109 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,38485,1676249635575 WAL splitting is done? wals=0, meta=false 2023-02-13 00:54:03,109 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,37023,1676249635625 WAL count=0, meta=false 2023-02-13 00:54:03,110 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,42839,1676249635601-splitting dir is empty, no logs to split. 2023-02-13 00:54:03,110 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,42839,1676249635601 WAL count=0, meta=false 2023-02-13 00:54:03,110 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42839,1676249635601 WAL splitting is done? wals=0, meta=false 2023-02-13 00:54:03,111 INFO [PEWorker-2] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,38485,1676249635575 failed, ignore...File hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,38485,1676249635575-splitting does not exist. 2023-02-13 00:54:03,112 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e95173dbc5ee3e5a46e6e44669d98b1f, ASSIGN}, {pid=24, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=4534c163be465f052ed9a3af03edbb95, ASSIGN}, {pid=25, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=e0dbd631ed14dc0143960d5a5baf5b84, ASSIGN}] 2023-02-13 00:54:03,112 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=0a98b367f319e93352882eb969190eed, ASSIGN}] 2023-02-13 00:54:03,113 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249635625-splitting dir is empty, no logs to split. 2023-02-13 00:54:03,113 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,37023,1676249635625 WAL count=0, meta=false 2023-02-13 00:54:03,113 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,37023,1676249635625 WAL splitting is done? wals=0, meta=false 2023-02-13 00:54:03,113 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e95173dbc5ee3e5a46e6e44669d98b1f, ASSIGN 2023-02-13 00:54:03,114 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=e0dbd631ed14dc0143960d5a5baf5b84, ASSIGN 2023-02-13 00:54:03,114 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=4534c163be465f052ed9a3af03edbb95, ASSIGN 2023-02-13 00:54:03,115 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:54:03,115 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=23, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e95173dbc5ee3e5a46e6e44669d98b1f, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,42839,1676249635601; forceNewPlan=false, retain=true 2023-02-13 00:54:03,115 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:54:03,115 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=0a98b367f319e93352882eb969190eed, ASSIGN 2023-02-13 00:54:03,115 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=e0dbd631ed14dc0143960d5a5baf5b84, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,42839,1676249635601; forceNewPlan=false, retain=true 2023-02-13 00:54:03,115 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42839,1676249635601 2023-02-13 00:54:03,116 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=4534c163be465f052ed9a3af03edbb95, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,42839,1676249635601; forceNewPlan=false, retain=true 2023-02-13 00:54:03,116 INFO [PEWorker-3] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,37023,1676249635625 failed, ignore...File hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/WALs/jenkins-hbase4.apache.org,37023,1676249635625-splitting does not exist. 2023-02-13 00:54:03,117 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38485,1676249635575 2023-02-13 00:54:03,117 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=0a98b367f319e93352882eb969190eed, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,38485,1676249635575; forceNewPlan=false, retain=true 2023-02-13 00:54:03,118 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,37023,1676249635625 after splitting done 2023-02-13 00:54:03,118 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,37023,1676249635625 from processing; numProcessing=2 2023-02-13 00:54:03,119 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,37023,1676249635625, splitWal=true, meta=false in 1.5150 sec 2023-02-13 00:54:03,265 INFO [jenkins-hbase4:35729] balancer.BaseLoadBalancer(1545): Reassigned 4 regions. 4 retained the pre-restart assignment. 2023-02-13 00:54:03,268 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=e95173dbc5ee3e5a46e6e44669d98b1f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:03,268 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e0dbd631ed14dc0143960d5a5baf5b84, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:03,268 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=0a98b367f319e93352882eb969190eed, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:03,268 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=4534c163be465f052ed9a3af03edbb95, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:03,268 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249643268"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249643268"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249643268"}]},"ts":"1676249643268"} 2023-02-13 00:54:03,268 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249643268"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249643268"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249643268"}]},"ts":"1676249643268"} 2023-02-13 00:54:03,268 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249643268"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249643268"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249643268"}]},"ts":"1676249643268"} 2023-02-13 00:54:03,268 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676249643268"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676249643268"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676249643268"}]},"ts":"1676249643268"} 2023-02-13 00:54:03,271 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=25, state=RUNNABLE; OpenRegionProcedure e0dbd631ed14dc0143960d5a5baf5b84, server=jenkins-hbase4.apache.org,42839,1676249641886}] 2023-02-13 00:54:03,272 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=26, state=RUNNABLE; OpenRegionProcedure 0a98b367f319e93352882eb969190eed, server=jenkins-hbase4.apache.org,38485,1676249641485}] 2023-02-13 00:54:03,273 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=23, state=RUNNABLE; OpenRegionProcedure e95173dbc5ee3e5a46e6e44669d98b1f, server=jenkins-hbase4.apache.org,42839,1676249641886}] 2023-02-13 00:54:03,274 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=24, state=RUNNABLE; OpenRegionProcedure 4534c163be465f052ed9a3af03edbb95, server=jenkins-hbase4.apache.org,42839,1676249641886}] 2023-02-13 00:54:03,426 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:03,426 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 00:54:03,428 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60468, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-13 00:54:03,429 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:54:03,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0dbd631ed14dc0143960d5a5baf5b84, NAME => 'restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:54:03,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:54:03,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:54:03,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7890): checking encryption for e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:54:03,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7893): checking classloading for e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:54:03,431 INFO [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:54:03,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:54:03,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0a98b367f319e93352882eb969190eed, NAME => 'restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:54:03,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 0a98b367f319e93352882eb969190eed 2023-02-13 00:54:03,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:54:03,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(7890): checking encryption for 0a98b367f319e93352882eb969190eed 2023-02-13 00:54:03,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(7893): checking classloading for 0a98b367f319e93352882eb969190eed 2023-02-13 00:54:03,433 DEBUG [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84/family 2023-02-13 00:54:03,433 DEBUG [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84/family 2023-02-13 00:54:03,433 INFO [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0dbd631ed14dc0143960d5a5baf5b84 columnFamilyName family 2023-02-13 00:54:03,434 INFO [StoreOpener-0a98b367f319e93352882eb969190eed-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 0a98b367f319e93352882eb969190eed 2023-02-13 00:54:03,434 INFO [StoreOpener-e0dbd631ed14dc0143960d5a5baf5b84-1] regionserver.HStore(310): Store=e0dbd631ed14dc0143960d5a5baf5b84/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:54:03,435 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:54:03,435 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7850): Opening region: {ENCODED => e95173dbc5ee3e5a46e6e44669d98b1f, NAME => 'hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:54:03,435 DEBUG [StoreOpener-0a98b367f319e93352882eb969190eed-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed/family 2023-02-13 00:54:03,435 DEBUG [StoreOpener-0a98b367f319e93352882eb969190eed-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed/family 2023-02-13 00:54:03,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:54:03,435 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:54:03,435 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:54:03,435 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7890): checking encryption for e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:54:03,435 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7893): checking classloading for e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:54:03,435 INFO [StoreOpener-0a98b367f319e93352882eb969190eed-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a98b367f319e93352882eb969190eed columnFamilyName family 2023-02-13 00:54:03,436 INFO [StoreOpener-0a98b367f319e93352882eb969190eed-1] regionserver.HStore(310): Store=0a98b367f319e93352882eb969190eed/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:54:03,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:54:03,437 INFO [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:54:03,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed 2023-02-13 00:54:03,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed 2023-02-13 00:54:03,440 DEBUG [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/info 2023-02-13 00:54:03,440 DEBUG [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/info 2023-02-13 00:54:03,440 INFO [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e95173dbc5ee3e5a46e6e44669d98b1f columnFamilyName info 2023-02-13 00:54:03,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(1054): writing seq id for 0a98b367f319e93352882eb969190eed 2023-02-13 00:54:03,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1054): writing seq id for e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:54:03,442 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(1071): Opened 0a98b367f319e93352882eb969190eed; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74739215, jitterRate=0.11370109021663666}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:54:03,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(964): Region open journal for 0a98b367f319e93352882eb969190eed: Running coprocessor pre-open hook at 1676249643433Writing region info on filesystem at 1676249643433Initializing all the Stores at 1676249643433Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249643433Cleaning up temporary data from old regions at 1676249643439 (+6 ms)Cleaning up detritus from prior splits at 1676249643439Running coprocessor post-open hooks at 1676249643442 (+3 ms)Region opened successfully at 1676249643442 2023-02-13 00:54:03,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1071): Opened e0dbd631ed14dc0143960d5a5baf5b84; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63378099, jitterRate=-0.05559273064136505}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:54:03,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(964): Region open journal for e0dbd631ed14dc0143960d5a5baf5b84: Running coprocessor pre-open hook at 1676249643430Writing region info on filesystem at 1676249643430Initializing all the Stores at 1676249643431 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249643431Cleaning up temporary data from old regions at 1676249643437 (+6 ms)Cleaning up detritus from prior splits at 1676249643437Running coprocessor post-open hooks at 1676249643443 (+6 ms)Region opened successfully at 1676249643443 2023-02-13 00:54:03,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84., pid=27, masterSystemTime=1676249643424 2023-02-13 00:54:03,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed., pid=28, masterSystemTime=1676249643426 2023-02-13 00:54:03,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:54:03,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:54:03,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:54:03,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4534c163be465f052ed9a3af03edbb95, NAME => 'restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.', STARTKEY => '', ENDKEY => ''} 2023-02-13 00:54:03,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:54:03,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 00:54:03,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7890): checking encryption for 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:54:03,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(7893): checking classloading for 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:54:03,451 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e0dbd631ed14dc0143960d5a5baf5b84, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:03,451 DEBUG [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] regionserver.HStore(539): loaded hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/info/f67aa219ce294fc9afb9b66e02ed8d3d 2023-02-13 00:54:03,451 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249643451"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249643451"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249643451"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249643451"}]},"ts":"1676249643451"} 2023-02-13 00:54:03,451 INFO [StoreOpener-e95173dbc5ee3e5a46e6e44669d98b1f-1] regionserver.HStore(310): Store=e95173dbc5ee3e5a46e6e44669d98b1f/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:54:03,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:54:03,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:38485-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:54:03,453 INFO [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:54:03,453 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:54:03,453 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=0a98b367f319e93352882eb969190eed, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:03,453 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676249643453"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249643453"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249643453"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249643453"}]},"ts":"1676249643453"} 2023-02-13 00:54:03,454 DEBUG [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95/family 2023-02-13 00:54:03,454 DEBUG [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95/family 2023-02-13 00:54:03,454 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:54:03,455 INFO [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4534c163be465f052ed9a3af03edbb95 columnFamilyName family 2023-02-13 00:54:03,455 INFO [StoreOpener-4534c163be465f052ed9a3af03edbb95-1] regionserver.HStore(310): Store=4534c163be465f052ed9a3af03edbb95/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 00:54:03,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95 2023-02-13 00:54:03,457 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=25 2023-02-13 00:54:03,457 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=25, state=SUCCESS; OpenRegionProcedure e0dbd631ed14dc0143960d5a5baf5b84, server=jenkins-hbase4.apache.org,42839,1676249641886 in 184 msec 2023-02-13 00:54:03,458 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=26 2023-02-13 00:54:03,458 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=26, state=SUCCESS; OpenRegionProcedure 0a98b367f319e93352882eb969190eed, server=jenkins-hbase4.apache.org,38485,1676249641485 in 184 msec 2023-02-13 00:54:03,459 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=e0dbd631ed14dc0143960d5a5baf5b84, ASSIGN in 345 msec 2023-02-13 00:54:03,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95 2023-02-13 00:54:03,459 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1054): writing seq id for e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:54:03,461 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1071): Opened e95173dbc5ee3e5a46e6e44669d98b1f; next sequenceid=10; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64123333, jitterRate=-0.04448787868022919}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:54:03,461 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(964): Region open journal for e95173dbc5ee3e5a46e6e44669d98b1f: Running coprocessor pre-open hook at 1676249643435Writing region info on filesystem at 1676249643435Initializing all the Stores at 1676249643436 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676249643436Cleaning up temporary data from old regions at 1676249643455 (+19 ms)Cleaning up detritus from prior splits at 1676249643456 (+1 ms)Running coprocessor post-open hooks at 1676249643461 (+5 ms)Region opened successfully at 1676249643461 2023-02-13 00:54:03,462 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f., pid=29, masterSystemTime=1676249643424 2023-02-13 00:54:03,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1054): writing seq id for 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:54:03,464 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-02-13 00:54:03,464 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,38485,1676249635575 after splitting done 2023-02-13 00:54:03,464 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,38485,1676249635575 from processing; numProcessing=1 2023-02-13 00:54:03,464 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=0a98b367f319e93352882eb969190eed, ASSIGN in 346 msec 2023-02-13 00:54:03,464 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:54:03,464 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:42839-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:54:03,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1071): Opened 4534c163be465f052ed9a3af03edbb95; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66038857, jitterRate=-0.01594434678554535}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 00:54:03,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(964): Region open journal for 4534c163be465f052ed9a3af03edbb95: Running coprocessor pre-open hook at 1676249643451Writing region info on filesystem at 1676249643451Initializing all the Stores at 1676249643452 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676249643452Cleaning up temporary data from old regions at 1676249643460 (+8 ms)Cleaning up detritus from prior splits at 1676249643460Running coprocessor post-open hooks at 1676249643465 (+5 ms)Region opened successfully at 1676249643465 2023-02-13 00:54:03,465 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=e95173dbc5ee3e5a46e6e44669d98b1f, regionState=OPEN, openSeqNum=10, regionLocation=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:03,465 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676249643465"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249643465"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249643465"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249643465"}]},"ts":"1676249643465"} 2023-02-13 00:54:03,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95., pid=30, masterSystemTime=1676249643424 2023-02-13 00:54:03,466 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,38485,1676249635575, splitWal=true, meta=false in 1.8580 sec 2023-02-13 00:54:03,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:54:03,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:42839-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:54:03,469 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=4534c163be465f052ed9a3af03edbb95, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:03,469 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676249643469"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676249643469"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676249643469"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676249643469"}]},"ts":"1676249643469"} 2023-02-13 00:54:03,469 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=23 2023-02-13 00:54:03,469 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=23, state=SUCCESS; OpenRegionProcedure e95173dbc5ee3e5a46e6e44669d98b1f, server=jenkins-hbase4.apache.org,42839,1676249641886 in 194 msec 2023-02-13 00:54:03,471 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e95173dbc5ee3e5a46e6e44669d98b1f, ASSIGN in 357 msec 2023-02-13 00:54:03,472 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=24 2023-02-13 00:54:03,473 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=24, state=SUCCESS; OpenRegionProcedure 4534c163be465f052ed9a3af03edbb95, server=jenkins-hbase4.apache.org,42839,1676249641886 in 196 msec 2023-02-13 00:54:03,475 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=20 2023-02-13 00:54:03,475 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,42839,1676249635601 after splitting done 2023-02-13 00:54:03,475 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=4534c163be465f052ed9a3af03edbb95, ASSIGN in 361 msec 2023-02-13 00:54:03,475 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,42839,1676249635601 from processing; numProcessing=0 2023-02-13 00:54:03,476 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,42839,1676249635601, splitWal=true, meta=true in 1.8670 sec 2023-02-13 00:54:03,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 00:54:03,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35729] ipc.CallRunner(144): callId: 58 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:43550 deadline: 1676249703488, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 00:54:04,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-13 00:54:04,115 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 00:54:04,118 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 00:54:04,118 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.641sec 2023-02-13 00:54:04,118 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 00:54:04,118 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 00:54:04,118 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 00:54:04,118 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35729,1676249641454-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 00:54:04,118 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35729,1676249641454-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 00:54:04,121 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 00:54:04,501 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:54:04,508 DEBUG [hconnection-0x429f910b-shared-pool-9] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 00:54:04,510 INFO [RS-EventLoopGroup-14-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60482, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 00:54:04,511 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:54:04,517 INFO [Listener at localhost/35605] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 00:54:04,517 INFO [Listener at localhost/35605] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 00:54:04,520 INFO [Listener at localhost/35605] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 00:54:04,520 INFO [Listener at localhost/35605] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => e95173dbc5ee3e5a46e6e44669d98b1f, NAME => 'hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,42839,1676249635601, currentServer=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:04,520 INFO [Listener at localhost/35605] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => 4534c163be465f052ed9a3af03edbb95, NAME => 'restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,42839,1676249635601, currentServer=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:04,520 INFO [Listener at localhost/35605] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => e0dbd631ed14dc0143960d5a5baf5b84, NAME => 'restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,42839,1676249635601, currentServer=jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:04,520 INFO [Listener at localhost/35605] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => 0a98b367f319e93352882eb969190eed, NAME => 'restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,38485,1676249635575, currentServer=jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:04,520 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 00:54:04,520 INFO [Listener at localhost/35605] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 00:54:04,521 DEBUG [Listener at localhost/35605] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ea9bbe5 to 127.0.0.1:57021 2023-02-13 00:54:04,521 DEBUG [Listener at localhost/35605] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:54:04,521 DEBUG [Listener at localhost/35605] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 00:54:04,521 DEBUG [Listener at localhost/35605] util.JVMClusterUtil(257): Found active master hash=864949625, stopped=false 2023-02-13 00:54:04,521 INFO [Listener at localhost/35605] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:04,523 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 00:54:04,523 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 00:54:04,524 INFO [Listener at localhost/35605] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 00:54:04,523 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 00:54:04,524 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 00:54:04,526 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:54:04,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:54:04,526 DEBUG [Listener at localhost/35605] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6465eb61 to 127.0.0.1:57021 2023-02-13 00:54:04,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:54:04,526 DEBUG [Listener at localhost/35605] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:54:04,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:54:04,526 INFO [Listener at localhost/35605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38485,1676249641485' ***** 2023-02-13 00:54:04,526 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1065): Closing user regions 2023-02-13 00:54:04,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 00:54:04,526 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(3304): Received CLOSE for 0a98b367f319e93352882eb969190eed 2023-02-13 00:54:04,526 INFO [Listener at localhost/35605] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 00:54:04,527 INFO [Listener at localhost/35605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42839,1676249641886' ***** 2023-02-13 00:54:04,527 INFO [Listener at localhost/35605] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 00:54:04,527 INFO [Listener at localhost/35605] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37023,1676249641971' ***** 2023-02-13 00:54:04,527 INFO [Listener at localhost/35605] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 00:54:04,527 INFO [RS:4;jenkins-hbase4:42839] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:54:04,527 INFO [RS:5;jenkins-hbase4:37023] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:54:04,527 INFO [RS:4;jenkins-hbase4:42839] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:54:04,527 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:54:04,527 INFO [RS:4;jenkins-hbase4:42839] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:54:04,528 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:54:04,528 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(3304): Received CLOSE for 4534c163be465f052ed9a3af03edbb95 2023-02-13 00:54:04,527 INFO [RS:5;jenkins-hbase4:37023] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:54:04,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(1603): Closing 0a98b367f319e93352882eb969190eed, disabling compactions & flushes 2023-02-13 00:54:04,529 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(3304): Received CLOSE for e0dbd631ed14dc0143960d5a5baf5b84 2023-02-13 00:54:04,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:54:04,529 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(3304): Received CLOSE for e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:54:04,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:54:04,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1603): Closing 4534c163be465f052ed9a3af03edbb95, disabling compactions & flushes 2023-02-13 00:54:04,529 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:04,529 INFO [RS:3;jenkins-hbase4:38485] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 00:54:04,529 DEBUG [RS:4;jenkins-hbase4:42839] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x265aa063 to 127.0.0.1:57021 2023-02-13 00:54:04,529 INFO [RS:3;jenkins-hbase4:38485] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 00:54:04,529 INFO [RS:5;jenkins-hbase4:37023] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:54:04,530 INFO [RS:3;jenkins-hbase4:38485] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 00:54:04,529 DEBUG [RS:4;jenkins-hbase4:42839] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:54:04,529 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 00:54:04,530 INFO [RS:4;jenkins-hbase4:42839] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:54:04,530 INFO [RS:4;jenkins-hbase4:42839] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:54:04,530 INFO [RS:4;jenkins-hbase4:42839] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:54:04,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:54:04,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. after waiting 0 ms 2023-02-13 00:54:04,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:54:04,530 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 00:54:04,530 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:04,530 INFO [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:04,530 DEBUG [RS:3;jenkins-hbase4:38485] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6424be94 to 127.0.0.1:57021 2023-02-13 00:54:04,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. after waiting 0 ms 2023-02-13 00:54:04,531 DEBUG [RS:3;jenkins-hbase4:38485] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:54:04,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:54:04,531 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 00:54:04,531 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 00:54:04,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:54:04,531 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1475): Waiting on 4 regions to close 2023-02-13 00:54:04,531 DEBUG [RS:5;jenkins-hbase4:37023] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65a5bbbe to 127.0.0.1:57021 2023-02-13 00:54:04,531 DEBUG [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1479): Online Regions={4534c163be465f052ed9a3af03edbb95=restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95., e0dbd631ed14dc0143960d5a5baf5b84=restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84., e95173dbc5ee3e5a46e6e44669d98b1f=hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 00:54:04,531 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 00:54:04,532 DEBUG [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1505): Waiting on 1588230740, 4534c163be465f052ed9a3af03edbb95, e0dbd631ed14dc0143960d5a5baf5b84, e95173dbc5ee3e5a46e6e44669d98b1f 2023-02-13 00:54:04,531 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1479): Online Regions={0a98b367f319e93352882eb969190eed=restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed.} 2023-02-13 00:54:04,532 DEBUG [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1505): Waiting on 0a98b367f319e93352882eb969190eed 2023-02-13 00:54:04,532 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 00:54:04,532 DEBUG [RS:5;jenkins-hbase4:37023] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:54:04,532 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 00:54:04,532 INFO [regionserver/jenkins-hbase4:37023.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:54:04,532 INFO [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37023,1676249641971; all regions closed. 2023-02-13 00:54:04,532 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 00:54:04,533 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=3.81 KB heapSize=6.91 KB 2023-02-13 00:54:04,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableThree/4534c163be465f052ed9a3af03edbb95/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 00:54:04,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableTwo/0a98b367f319e93352882eb969190eed/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 00:54:04,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1837): Closed restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:54:04,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1557): Region close journal for 4534c163be465f052ed9a3af03edbb95: Waiting for close lock at 1676249644529Running coprocessor pre-close hooks at 1676249644529Disabling compacts and flushes for region at 1676249644529Disabling writes for close at 1676249644531 (+2 ms)Writing region close event to WAL at 1676249644532 (+1 ms)Running coprocessor post-close hooks at 1676249644538 (+6 ms)Closed at 1676249644538 2023-02-13 00:54:04,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676249638109.4534c163be465f052ed9a3af03edbb95. 2023-02-13 00:54:04,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:54:04,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1603): Closing e0dbd631ed14dc0143960d5a5baf5b84, disabling compactions & flushes 2023-02-13 00:54:04,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485-0] regionserver.HRegion(1557): Region close journal for 0a98b367f319e93352882eb969190eed: Waiting for close lock at 1676249644527Running coprocessor pre-close hooks at 1676249644527Disabling compacts and flushes for region at 1676249644527Disabling writes for close at 1676249644530 (+3 ms)Writing region close event to WAL at 1676249644531 (+1 ms)Running coprocessor post-close hooks at 1676249644539 (+8 ms)Closed at 1676249644539 2023-02-13 00:54:04,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:54:04,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:54:04,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:38485-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676249637495.0a98b367f319e93352882eb969190eed. 2023-02-13 00:54:04,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. after waiting 0 ms 2023-02-13 00:54:04,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:54:04,541 DEBUG [RS:5;jenkins-hbase4:37023] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs 2023-02-13 00:54:04,541 INFO [RS:5;jenkins-hbase4:37023] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37023%2C1676249641971:(num 1676249642157) 2023-02-13 00:54:04,541 DEBUG [RS:5;jenkins-hbase4:37023] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:54:04,542 INFO [RS:5;jenkins-hbase4:37023] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:54:04,542 INFO [regionserver/jenkins-hbase4:42839.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:54:04,542 INFO [RS:5;jenkins-hbase4:37023] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:37023 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 00:54:04,543 INFO [RS:5;jenkins-hbase4:37023] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:54:04,543 INFO [regionserver/jenkins-hbase4:37023.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:54:04,543 INFO [RS:5;jenkins-hbase4:37023] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:54:04,543 INFO [RS:5;jenkins-hbase4:37023] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:54:04,544 INFO [RS:5;jenkins-hbase4:37023] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37023 2023-02-13 00:54:04,551 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:04,551 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:04,551 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:04,551 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:04,551 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37023,1676249641971 2023-02-13 00:54:04,551 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:04,551 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@757e4e48 rejected from java.util.concurrent.ThreadPoolExecutor@7141c43f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:04,552 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:04,552 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2b65663 rejected from java.util.concurrent.ThreadPoolExecutor@7141c43f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:04,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/default/restartTableOne/e0dbd631ed14dc0143960d5a5baf5b84/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 00:54:04,554 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37023,1676249641971] 2023-02-13 00:54:04,554 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37023,1676249641971; numProcessing=1 2023-02-13 00:54:04,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1837): Closed restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:54:04,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1557): Region close journal for e0dbd631ed14dc0143960d5a5baf5b84: Waiting for close lock at 1676249644539Running coprocessor pre-close hooks at 1676249644539Disabling compacts and flushes for region at 1676249644539Disabling writes for close at 1676249644541 (+2 ms)Writing region close event to WAL at 1676249644541Running coprocessor post-close hooks at 1676249644555 (+14 ms)Closed at 1676249644555 2023-02-13 00:54:04,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676249636874.e0dbd631ed14dc0143960d5a5baf5b84. 2023-02-13 00:54:04,555 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37023,1676249641971 already deleted, retry=false 2023-02-13 00:54:04,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1603): Closing e95173dbc5ee3e5a46e6e44669d98b1f, disabling compactions & flushes 2023-02-13 00:54:04,555 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37023,1676249641971 expired; onlineServers=2 2023-02-13 00:54:04,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:54:04,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:54:04,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. after waiting 0 ms 2023-02-13 00:54:04,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:54:04,557 INFO [regionserver/jenkins-hbase4:38485.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:54:04,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/namespace/e95173dbc5ee3e5a46e6e44669d98b1f/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=9 2023-02-13 00:54:04,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:54:04,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1557): Region close journal for e95173dbc5ee3e5a46e6e44669d98b1f: Waiting for close lock at 1676249644555Running coprocessor pre-close hooks at 1676249644555Disabling compacts and flushes for region at 1676249644555Disabling writes for close at 1676249644555Writing region close event to WAL at 1676249644558 (+3 ms)Running coprocessor post-close hooks at 1676249644564 (+6 ms)Closed at 1676249644564 2023-02-13 00:54:04,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:42839-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676249636296.e95173dbc5ee3e5a46e6e44669d98b1f. 2023-02-13 00:54:04,567 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.81 KB at sequenceid=38 (bloomFilter=false), to=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/.tmp/info/5742bbb0b76b42bbaa9517ca59b50722 2023-02-13 00:54:04,573 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/.tmp/info/5742bbb0b76b42bbaa9517ca59b50722 as hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info/5742bbb0b76b42bbaa9517ca59b50722 2023-02-13 00:54:04,579 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HStore(1080): Added hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/info/5742bbb0b76b42bbaa9517ca59b50722, entries=32, sequenceid=38, filesize=8.5 K 2023-02-13 00:54:04,580 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(2947): Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 47ms, sequenceid=38, compaction requested=false 2023-02-13 00:54:04,587 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/data/hbase/meta/1588230740/recovered.edits/41.seqid, newMaxSeqId=41, maxSeqId=27 2023-02-13 00:54:04,588 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 00:54:04,588 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 00:54:04,588 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676249644531Running coprocessor pre-close hooks at 1676249644531Disabling compacts and flushes for region at 1676249644531Disabling writes for close at 1676249644532 (+1 ms)Obtaining lock to block concurrent updates at 1676249644533 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676249644533Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=3900, getHeapSize=7032, getOffHeapSize=0, getCellsCount=32 at 1676249644534 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676249644534Flushing 1588230740/info: creating writer at 1676249644534Flushing 1588230740/info: appending metadata at 1676249644542 (+8 ms)Flushing 1588230740/info: closing flushed file at 1676249644542Flushing 1588230740/info: reopening flushed file at 1676249644574 (+32 ms)Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 47ms, sequenceid=38, compaction requested=false at 1676249644580 (+6 ms)Writing region close event to WAL at 1676249644584 (+4 ms)Running coprocessor post-close hooks at 1676249644588 (+4 ms)Closed at 1676249644588 2023-02-13 00:54:04,588 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:42839-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 00:54:04,723 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:54:04,723 INFO [RS:5;jenkins-hbase4:37023] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37023,1676249641971; zookeeper connection closed. 2023-02-13 00:54:04,723 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7a2218d7 rejected from java.util.concurrent.ThreadPoolExecutor@7141c43f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:04,723 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:37023-0x1004b2b846b000f, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:54:04,723 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4220fee2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4220fee2 2023-02-13 00:54:04,723 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64d9ea9c rejected from java.util.concurrent.ThreadPoolExecutor@7141c43f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:04,732 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42839,1676249641886; all regions closed. 2023-02-13 00:54:04,732 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38485,1676249641485; all regions closed. 2023-02-13 00:54:04,738 DEBUG [RS:3;jenkins-hbase4:38485] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs 2023-02-13 00:54:04,738 INFO [RS:3;jenkins-hbase4:38485] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38485%2C1676249641485:(num 1676249641981) 2023-02-13 00:54:04,738 DEBUG [RS:3;jenkins-hbase4:38485] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:54:04,738 INFO [RS:3;jenkins-hbase4:38485] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:54:04,738 INFO [RS:3;jenkins-hbase4:38485] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:38485 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 00:54:04,738 INFO [RS:3;jenkins-hbase4:38485] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 00:54:04,738 INFO [RS:3;jenkins-hbase4:38485] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 00:54:04,738 INFO [RS:3;jenkins-hbase4:38485] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 00:54:04,738 INFO [regionserver/jenkins-hbase4:38485.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:54:04,738 DEBUG [RS:4;jenkins-hbase4:42839] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs 2023-02-13 00:54:04,739 INFO [RS:4;jenkins-hbase4:42839] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42839%2C1676249641886.meta:.meta(num 1676249642239) 2023-02-13 00:54:04,739 INFO [RS:3;jenkins-hbase4:38485] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38485 2023-02-13 00:54:04,742 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:04,742 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:04,742 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38485,1676249641485 2023-02-13 00:54:04,743 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@71b4dfdb rejected from java.util.concurrent.ThreadPoolExecutor@758b4535[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:04,745 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38485,1676249641485] 2023-02-13 00:54:04,745 DEBUG [RS:4;jenkins-hbase4:42839] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/oldWALs 2023-02-13 00:54:04,745 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38485,1676249641485; numProcessing=2 2023-02-13 00:54:04,745 INFO [RS:4;jenkins-hbase4:42839] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42839%2C1676249641886:(num 1676249642065) 2023-02-13 00:54:04,745 DEBUG [RS:4;jenkins-hbase4:42839] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:54:04,745 INFO [RS:4;jenkins-hbase4:42839] regionserver.LeaseManager(133): Closed leases 2023-02-13 00:54:04,745 INFO [RS:4;jenkins-hbase4:42839] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:42839 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 00:54:04,745 INFO [regionserver/jenkins-hbase4:42839.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:54:04,746 INFO [RS:4;jenkins-hbase4:42839] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42839 2023-02-13 00:54:04,746 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38485,1676249641485 already deleted, retry=false 2023-02-13 00:54:04,746 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38485,1676249641485 expired; onlineServers=1 2023-02-13 00:54:04,747 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 00:54:04,747 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42839,1676249641886 2023-02-13 00:54:04,747 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@25988130 rejected from java.util.concurrent.ThreadPoolExecutor@21a7015b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:04,748 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42839,1676249641886] 2023-02-13 00:54:04,749 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42839,1676249641886; numProcessing=3 2023-02-13 00:54:04,751 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42839,1676249641886 already deleted, retry=false 2023-02-13 00:54:04,751 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42839,1676249641886 expired; onlineServers=0 2023-02-13 00:54:04,751 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35729,1676249641454' ***** 2023-02-13 00:54:04,751 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 00:54:04,753 DEBUG [M:1;jenkins-hbase4:35729] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37b187d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 00:54:04,753 INFO [M:1;jenkins-hbase4:35729] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35729,1676249641454 2023-02-13 00:54:04,753 INFO [M:1;jenkins-hbase4:35729] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35729,1676249641454; all regions closed. 2023-02-13 00:54:04,753 DEBUG [M:1;jenkins-hbase4:35729] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 00:54:04,753 DEBUG [M:1;jenkins-hbase4:35729] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 00:54:04,753 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 00:54:04,753 DEBUG [M:1;jenkins-hbase4:35729] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 00:54:04,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249641660] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676249641660,5,FailOnTimeoutGroup] 2023-02-13 00:54:04,753 INFO [M:1;jenkins-hbase4:35729] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 00:54:04,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249641653] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676249641653,5,FailOnTimeoutGroup] 2023-02-13 00:54:04,754 INFO [M:1;jenkins-hbase4:35729] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 00:54:04,754 INFO [M:1;jenkins-hbase4:35729] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 00:54:04,754 DEBUG [M:1;jenkins-hbase4:35729] master.HMaster(1502): Stopping service threads 2023-02-13 00:54:04,754 INFO [M:1;jenkins-hbase4:35729] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 00:54:04,755 ERROR [M:1;jenkins-hbase4:35729] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-13 00:54:04,755 INFO [M:1;jenkins-hbase4:35729] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 00:54:04,755 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 00:54:04,755 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 00:54:04,755 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 00:54:04,756 DEBUG [M:1;jenkins-hbase4:35729] zookeeper.ZKUtil(398): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 00:54:04,756 WARN [M:1;jenkins-hbase4:35729] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 00:54:04,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 00:54:04,756 INFO [M:1;jenkins-hbase4:35729] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 00:54:04,756 INFO [M:1;jenkins-hbase4:35729] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 00:54:04,756 DEBUG [M:1;jenkins-hbase4:35729] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 00:54:04,756 INFO [M:1;jenkins-hbase4:35729] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:54:04,756 DEBUG [M:1;jenkins-hbase4:35729] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:54:04,756 DEBUG [M:1;jenkins-hbase4:35729] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 00:54:04,756 DEBUG [M:1;jenkins-hbase4:35729] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:54:04,756 INFO [M:1;jenkins-hbase4:35729] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=38.74 KB heapSize=47.95 KB 2023-02-13 00:54:04,768 INFO [M:1;jenkins-hbase4:35729] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.74 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ca50b6716abe47edb3ed6a7b19bee017 2023-02-13 00:54:04,773 DEBUG [M:1;jenkins-hbase4:35729] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ca50b6716abe47edb3ed6a7b19bee017 as hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ca50b6716abe47edb3ed6a7b19bee017 2023-02-13 00:54:04,778 INFO [M:1;jenkins-hbase4:35729] regionserver.HStore(1080): Added hdfs://localhost:44257/user/jenkins/test-data/0c590265-ae9a-15e3-ed7a-2b0f4d4a5ebc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ca50b6716abe47edb3ed6a7b19bee017, entries=13, sequenceid=246, filesize=6.2 K 2023-02-13 00:54:04,779 INFO [M:1;jenkins-hbase4:35729] regionserver.HRegion(2947): Finished flush of dataSize ~38.74 KB/39667, heapSize ~47.93 KB/49080, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 23ms, sequenceid=246, compaction requested=false 2023-02-13 00:54:04,783 INFO [M:1;jenkins-hbase4:35729] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 00:54:04,783 DEBUG [M:1;jenkins-hbase4:35729] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676249644756Disabling compacts and flushes for region at 1676249644756Disabling writes for close at 1676249644756Obtaining lock to block concurrent updates at 1676249644757 (+1 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676249644757Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=39667, getHeapSize=49080, getOffHeapSize=0, getCellsCount=121 at 1676249644757Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676249644757Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676249644758 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676249644761 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676249644761Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676249644774 (+13 ms)Finished flush of dataSize ~38.74 KB/39667, heapSize ~47.93 KB/49080, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 23ms, sequenceid=246, compaction requested=false at 1676249644779 (+5 ms)Writing region close event to WAL at 1676249644783 (+4 ms)Closed at 1676249644783 2023-02-13 00:54:04,786 INFO [M:1;jenkins-hbase4:35729] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 00:54:04,786 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 00:54:04,786 INFO [M:1;jenkins-hbase4:35729] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35729 2023-02-13 00:54:04,788 DEBUG [M:1;jenkins-hbase4:35729] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,35729,1676249641454 already deleted, retry=false 2023-02-13 00:54:05,224 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:54:05,224 INFO [M:1;jenkins-hbase4:35729] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35729,1676249641454; zookeeper connection closed. 2023-02-13 00:54:05,224 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4cc796e5 rejected from java.util.concurrent.ThreadPoolExecutor@48f80ba5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:05,225 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): master:35729-0x1004b2b846b0009, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:54:05,225 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4cbd437c rejected from java.util.concurrent.ThreadPoolExecutor@48f80ba5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:05,324 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:54:05,324 INFO [RS:4;jenkins-hbase4:42839] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42839,1676249641886; zookeeper connection closed. 2023-02-13 00:54:05,324 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@34fbbb86 rejected from java.util.concurrent.ThreadPoolExecutor@21a7015b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:05,325 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:42839-0x1004b2b846b000d, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:54:05,325 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@33a6aa7f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@33a6aa7f 2023-02-13 00:54:05,325 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c3f431b rejected from java.util.concurrent.ThreadPoolExecutor@21a7015b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:05,425 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:54:05,425 INFO [RS:3;jenkins-hbase4:38485] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38485,1676249641485; zookeeper connection closed. 2023-02-13 00:54:05,425 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@566ec4b2 rejected from java.util.concurrent.ThreadPoolExecutor@758b4535[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:05,425 DEBUG [Listener at localhost/35605-EventThread] zookeeper.ZKWatcher(600): regionserver:38485-0x1004b2b846b000a, quorum=127.0.0.1:57021, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 00:54:05,425 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3b7b68] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3b7b68 2023-02-13 00:54:05,425 ERROR [Listener at localhost/35605-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2e7ecc55 rejected from java.util.concurrent.ThreadPoolExecutor@758b4535[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 00:54:05,426 INFO [Listener at localhost/35605] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 6 regionserver(s) complete 2023-02-13 00:54:05,426 WARN [Listener at localhost/35605] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 00:54:05,430 INFO [Listener at localhost/35605] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 00:54:05,535 WARN [BP-816828255-172.31.14.131-1676249634767 heartbeating to localhost/127.0.0.1:44257] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 00:54:05,536 WARN [BP-816828255-172.31.14.131-1676249634767 heartbeating to localhost/127.0.0.1:44257] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-816828255-172.31.14.131-1676249634767 (Datanode Uuid dd8bba80-6bef-4cfb-9080-9a464876a28f) service to localhost/127.0.0.1:44257 2023-02-13 00:54:05,537 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456/dfs/data/data5/current/BP-816828255-172.31.14.131-1676249634767] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:54:05,537 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456/dfs/data/data6/current/BP-816828255-172.31.14.131-1676249634767] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:54:05,538 WARN [Listener at localhost/35605] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 00:54:05,541 INFO [Listener at localhost/35605] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 00:54:05,647 WARN [BP-816828255-172.31.14.131-1676249634767 heartbeating to localhost/127.0.0.1:44257] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 00:54:05,647 WARN [BP-816828255-172.31.14.131-1676249634767 heartbeating to localhost/127.0.0.1:44257] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-816828255-172.31.14.131-1676249634767 (Datanode Uuid 378f2e60-c4c4-4671-aa81-d1af68de1789) service to localhost/127.0.0.1:44257 2023-02-13 00:54:05,647 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456/dfs/data/data3/current/BP-816828255-172.31.14.131-1676249634767] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:54:05,648 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456/dfs/data/data4/current/BP-816828255-172.31.14.131-1676249634767] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:54:05,649 WARN [Listener at localhost/35605] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 00:54:05,652 INFO [Listener at localhost/35605] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 00:54:05,755 WARN [BP-816828255-172.31.14.131-1676249634767 heartbeating to localhost/127.0.0.1:44257] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 00:54:05,755 WARN [BP-816828255-172.31.14.131-1676249634767 heartbeating to localhost/127.0.0.1:44257] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-816828255-172.31.14.131-1676249634767 (Datanode Uuid 8cf3b173-72eb-4183-a548-1c31563c948a) service to localhost/127.0.0.1:44257 2023-02-13 00:54:05,756 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456/dfs/data/data1/current/BP-816828255-172.31.14.131-1676249634767] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:54:05,756 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aaf14319-a230-8e62-20ab-56747bd04f01/cluster_a54a1623-ffea-23b7-f840-07a376511456/dfs/data/data2/current/BP-816828255-172.31.14.131-1676249634767] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 00:54:05,765 INFO [Listener at localhost/35605] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 00:54:05,882 INFO [Listener at localhost/35605] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 00:54:05,912 INFO [Listener at localhost/35605] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-13 00:54:05,926 INFO [Listener at localhost/35605] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=157 (was 100) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:44257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.9@localhost:44257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:44257 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:44257 from jenkins.hfs.10 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.10@localhost:44257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.11@localhost:44257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:44257 from jenkins.hfs.11 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/35605 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.7@localhost:44257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:38485.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:44257 from jenkins.hfs.7 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.8@localhost:44257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:44257 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:44257 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:44257 from jenkins.hfs.9 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:37023.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: LeaseRenewer:jenkins.hfs.6@localhost:44257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:42839.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:44257 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:44257 from jenkins.hfs.6 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-16-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (1422729484) connection to localhost/127.0.0.1:44257 from jenkins.hfs.8 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=581 (was 507) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=257 (was 277), ProcessCount=168 (was 168), AvailableMemoryMB=3847 (was 4090)