2023-02-13 16:53:33,573 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart timeout: 13 mins 2023-02-13 16:53:33,575 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestartSplitWithoutZk timeout: 13 mins 2023-02-13 16:53:33,613 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=7, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=357, ProcessCount=168, AvailableMemoryMB=2924 2023-02-13 16:53:33,865 DEBUG [Time-limited test] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f 2023-02-13 16:53:33,867 INFO [Time-limited test] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-13 16:53:33,870 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 16:53:33,871 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48, deleteOnExit=true 2023-02-13 16:53:33,871 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 16:53:33,871 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/test.cache.data in system properties and HBase conf 2023-02-13 16:53:33,872 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 16:53:33,872 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/hadoop.log.dir in system properties and HBase conf 2023-02-13 16:53:33,872 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 16:53:33,873 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 16:53:33,873 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 16:53:33,980 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-13 16:53:34,394 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 16:53:34,399 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 16:53:34,400 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 16:53:34,400 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 16:53:34,400 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 16:53:34,401 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 16:53:34,401 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 16:53:34,402 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 16:53:34,402 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 16:53:34,403 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 16:53:34,403 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/nfs.dump.dir in system properties and HBase conf 2023-02-13 16:53:34,404 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/java.io.tmpdir in system properties and HBase conf 2023-02-13 16:53:34,404 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 16:53:34,405 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 16:53:34,405 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 16:53:34,999 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 16:53:35,003 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 16:53:35,293 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-13 16:53:35,460 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-13 16:53:35,485 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 16:53:35,524 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 16:53:35,557 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/java.io.tmpdir/Jetty_localhost_36073_hdfs____ktjgc3/webapp 2023-02-13 16:53:35,702 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36073 2023-02-13 16:53:35,722 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 16:53:35,722 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 16:53:36,292 WARN [Listener at localhost/39815] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 16:53:36,363 WARN [Listener at localhost/39815] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 16:53:36,384 WARN [Listener at localhost/39815] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 16:53:36,392 INFO [Listener at localhost/39815] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 16:53:36,400 INFO [Listener at localhost/39815] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/java.io.tmpdir/Jetty_localhost_43127_datanode____x3196m/webapp 2023-02-13 16:53:36,503 INFO [Listener at localhost/39815] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43127 2023-02-13 16:53:36,900 WARN [Listener at localhost/40073] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 16:53:36,915 WARN [Listener at localhost/40073] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 16:53:36,928 WARN [Listener at localhost/40073] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 16:53:36,931 INFO [Listener at localhost/40073] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 16:53:36,939 INFO [Listener at localhost/40073] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/java.io.tmpdir/Jetty_localhost_42405_datanode____oswo2/webapp 2023-02-13 16:53:37,074 INFO [Listener at localhost/40073] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42405 2023-02-13 16:53:37,092 WARN [Listener at localhost/36987] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 16:53:37,135 WARN [Listener at localhost/36987] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 16:53:37,144 WARN [Listener at localhost/36987] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 16:53:37,147 INFO [Listener at localhost/36987] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 16:53:37,164 INFO [Listener at localhost/36987] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/java.io.tmpdir/Jetty_localhost_38777_datanode____58ri61/webapp 2023-02-13 16:53:37,316 INFO [Listener at localhost/36987] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38777 2023-02-13 16:53:37,339 WARN [Listener at localhost/45541] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 16:53:37,586 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x30609e53e92b1690: Processing first storage report for DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a from datanode 32e6136a-e7c4-43a4-9f8d-381de6a24c22 2023-02-13 16:53:37,587 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x30609e53e92b1690: from storage DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a node DatanodeRegistration(127.0.0.1:41403, datanodeUuid=32e6136a-e7c4-43a4-9f8d-381de6a24c22, infoPort=40635, infoSecurePort=0, ipcPort=45541, storageInfo=lv=-57;cid=testClusterID;nsid=1915718588;c=1676307215080), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-13 16:53:37,588 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x74f85c4a9cdcfc30: Processing first storage report for DS-42509378-a03d-45f3-848b-61755295c521 from datanode e38a03ee-4b5e-4a0b-a0f9-95c3afcd0c1f 2023-02-13 16:53:37,588 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x74f85c4a9cdcfc30: from storage DS-42509378-a03d-45f3-848b-61755295c521 node DatanodeRegistration(127.0.0.1:34747, datanodeUuid=e38a03ee-4b5e-4a0b-a0f9-95c3afcd0c1f, infoPort=43569, infoSecurePort=0, ipcPort=36987, storageInfo=lv=-57;cid=testClusterID;nsid=1915718588;c=1676307215080), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:37,588 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc8ae5afe5b5eb9a9: Processing first storage report for DS-34034e11-090f-4563-b231-61dfc67b7a08 from datanode f56ddadc-7b09-4b7b-8277-059e3c8c86f8 2023-02-13 16:53:37,588 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc8ae5afe5b5eb9a9: from storage DS-34034e11-090f-4563-b231-61dfc67b7a08 node DatanodeRegistration(127.0.0.1:44649, datanodeUuid=f56ddadc-7b09-4b7b-8277-059e3c8c86f8, infoPort=42635, infoSecurePort=0, ipcPort=40073, storageInfo=lv=-57;cid=testClusterID;nsid=1915718588;c=1676307215080), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:37,588 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x30609e53e92b1690: Processing first storage report for DS-24c99ba3-5ddc-48a6-b9fb-8910d6f1762d from datanode 32e6136a-e7c4-43a4-9f8d-381de6a24c22 2023-02-13 16:53:37,588 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x30609e53e92b1690: from storage DS-24c99ba3-5ddc-48a6-b9fb-8910d6f1762d node DatanodeRegistration(127.0.0.1:41403, datanodeUuid=32e6136a-e7c4-43a4-9f8d-381de6a24c22, infoPort=40635, infoSecurePort=0, ipcPort=45541, storageInfo=lv=-57;cid=testClusterID;nsid=1915718588;c=1676307215080), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:37,589 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x74f85c4a9cdcfc30: Processing first storage report for DS-7c6fd0d5-6c05-4750-8e14-92e4c1cdfc2b from datanode e38a03ee-4b5e-4a0b-a0f9-95c3afcd0c1f 2023-02-13 16:53:37,589 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x74f85c4a9cdcfc30: from storage DS-7c6fd0d5-6c05-4750-8e14-92e4c1cdfc2b node DatanodeRegistration(127.0.0.1:34747, datanodeUuid=e38a03ee-4b5e-4a0b-a0f9-95c3afcd0c1f, infoPort=43569, infoSecurePort=0, ipcPort=36987, storageInfo=lv=-57;cid=testClusterID;nsid=1915718588;c=1676307215080), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-13 16:53:37,589 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc8ae5afe5b5eb9a9: Processing first storage report for DS-40f078ae-7b75-4433-8fae-574ad6163742 from datanode f56ddadc-7b09-4b7b-8277-059e3c8c86f8 2023-02-13 16:53:37,589 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc8ae5afe5b5eb9a9: from storage DS-40f078ae-7b75-4433-8fae-574ad6163742 node DatanodeRegistration(127.0.0.1:44649, datanodeUuid=f56ddadc-7b09-4b7b-8277-059e3c8c86f8, infoPort=42635, infoSecurePort=0, ipcPort=40073, storageInfo=lv=-57;cid=testClusterID;nsid=1915718588;c=1676307215080), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:37,782 DEBUG [Listener at localhost/45541] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f 2023-02-13 16:53:37,848 INFO [Listener at localhost/45541] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48/zookeeper_0, clientPort=59338, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 16:53:37,864 INFO [Listener at localhost/45541] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=59338 2023-02-13 16:53:37,875 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:37,878 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:38,561 INFO [Listener at localhost/45541] util.FSUtils(479): Created version file at hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2 with version=8 2023-02-13 16:53:38,561 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/hbase-staging 2023-02-13 16:53:38,890 INFO [Listener at localhost/45541] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 16:53:39,353 INFO [Listener at localhost/45541] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:39,389 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:39,389 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:39,390 INFO [Listener at localhost/45541] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:39,390 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:39,390 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:39,533 INFO [Listener at localhost/45541] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:39,613 DEBUG [Listener at localhost/45541] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-13 16:53:39,717 INFO [Listener at localhost/45541] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39843 2023-02-13 16:53:39,727 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:39,728 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:39,754 INFO [Listener at localhost/45541] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39843 connecting to ZooKeeper ensemble=127.0.0.1:59338 2023-02-13 16:53:39,802 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:398430x0, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:39,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39843-0x1004e9a276c0000 connected 2023-02-13 16:53:39,833 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:39,834 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:39,838 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:39,845 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39843 2023-02-13 16:53:39,846 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39843 2023-02-13 16:53:39,846 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39843 2023-02-13 16:53:39,847 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39843 2023-02-13 16:53:39,847 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39843 2023-02-13 16:53:39,853 INFO [Listener at localhost/45541] master.HMaster(439): hbase.rootdir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2, hbase.cluster.distributed=false 2023-02-13 16:53:39,928 INFO [Listener at localhost/45541] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:39,928 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:39,928 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:39,928 INFO [Listener at localhost/45541] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:39,928 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:39,928 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:39,933 INFO [Listener at localhost/45541] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:39,936 INFO [Listener at localhost/45541] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33173 2023-02-13 16:53:39,939 INFO [Listener at localhost/45541] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:39,945 DEBUG [Listener at localhost/45541] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:39,946 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:39,949 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:39,951 INFO [Listener at localhost/45541] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33173 connecting to ZooKeeper ensemble=127.0.0.1:59338 2023-02-13 16:53:39,959 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:331730x0, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:39,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33173-0x1004e9a276c0001 connected 2023-02-13 16:53:39,961 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:39,962 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:39,963 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:39,964 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33173 2023-02-13 16:53:39,969 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33173 2023-02-13 16:53:39,970 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33173 2023-02-13 16:53:39,970 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33173 2023-02-13 16:53:39,970 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33173 2023-02-13 16:53:39,984 INFO [Listener at localhost/45541] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:39,984 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:39,984 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:39,985 INFO [Listener at localhost/45541] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:39,985 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:39,985 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:39,986 INFO [Listener at localhost/45541] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:39,987 INFO [Listener at localhost/45541] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37285 2023-02-13 16:53:39,988 INFO [Listener at localhost/45541] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:39,989 DEBUG [Listener at localhost/45541] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:39,990 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:39,993 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:39,996 INFO [Listener at localhost/45541] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37285 connecting to ZooKeeper ensemble=127.0.0.1:59338 2023-02-13 16:53:40,000 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:372850x0, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:40,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37285-0x1004e9a276c0002 connected 2023-02-13 16:53:40,001 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:40,002 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:40,003 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:40,003 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37285 2023-02-13 16:53:40,003 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37285 2023-02-13 16:53:40,006 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37285 2023-02-13 16:53:40,007 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37285 2023-02-13 16:53:40,007 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37285 2023-02-13 16:53:40,019 INFO [Listener at localhost/45541] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:40,019 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:40,019 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:40,020 INFO [Listener at localhost/45541] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:40,020 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:40,020 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:40,020 INFO [Listener at localhost/45541] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:40,021 INFO [Listener at localhost/45541] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45145 2023-02-13 16:53:40,022 INFO [Listener at localhost/45541] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:40,023 DEBUG [Listener at localhost/45541] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:40,024 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:40,025 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:40,026 INFO [Listener at localhost/45541] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45145 connecting to ZooKeeper ensemble=127.0.0.1:59338 2023-02-13 16:53:40,031 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:451450x0, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:40,031 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:451450x0, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:40,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:45145-0x1004e9a276c0003 connected 2023-02-13 16:53:40,032 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:40,033 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:40,036 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45145 2023-02-13 16:53:40,036 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45145 2023-02-13 16:53:40,037 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45145 2023-02-13 16:53:40,037 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45145 2023-02-13 16:53:40,037 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45145 2023-02-13 16:53:40,040 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:40,049 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 16:53:40,051 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:40,067 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:40,068 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:40,068 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:40,068 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:40,069 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:40,069 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:40,071 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39843,1676307218728 from backup master directory 2023-02-13 16:53:40,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:40,075 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:40,075 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 16:53:40,076 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:40,076 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:40,079 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-13 16:53:40,081 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-13 16:53:40,179 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/hbase.id with ID: 1a36896f-674b-48e6-9f6c-5070f3bef7ac 2023-02-13 16:53:40,226 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:40,242 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:40,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x189a24d4 to 127.0.0.1:59338 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:40,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21431242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:40,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:40,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 16:53:40,375 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-13 16:53:40,375 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-13 16:53:40,377 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:40,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:40,382 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:40,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store-tmp 2023-02-13 16:53:40,492 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:40,493 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 16:53:40,493 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:40,493 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:40,493 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 16:53:40,494 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:40,494 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:40,494 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676307220492Disabling compacts and flushes for region at 1676307220493 (+1 ms)Disabling writes for close at 1676307220494 (+1 ms)Writing region close event to WAL at 1676307220494Closed at 1676307220494 2023-02-13 16:53:40,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:40,516 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39843%2C1676307218728, suffix=, logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,39843,1676307218728, archiveDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/oldWALs, maxLogs=10 2023-02-13 16:53:40,589 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK] 2023-02-13 16:53:40,589 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK] 2023-02-13 16:53:40,589 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK] 2023-02-13 16:53:40,598 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:40,675 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,39843,1676307218728/jenkins-hbase4.apache.org%2C39843%2C1676307218728.1676307220526 2023-02-13 16:53:40,676 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK], DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK], DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK]] 2023-02-13 16:53:40,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:40,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:40,680 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:40,681 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:40,744 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:40,754 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 16:53:40,786 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 16:53:40,801 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:40,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:40,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:40,832 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:40,837 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:40,838 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58738508, jitterRate=-0.12472802400588989}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:40,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676307220707Initializing all the Stores at 1676307220709 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307220710 (+1 ms)Cleaning up temporary data from old regions at 1676307220822 (+112 ms)Cleaning up detritus from prior splits at 1676307220823 (+1 ms)Region opened successfully at 1676307220839 (+16 ms) 2023-02-13 16:53:40,840 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 16:53:40,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 16:53:40,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 16:53:40,866 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 16:53:40,868 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-13 16:53:40,905 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 36 msec 2023-02-13 16:53:40,905 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 16:53:40,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 16:53:40,945 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 16:53:40,970 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 16:53:40,974 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 16:53:40,975 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 16:53:40,980 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 16:53:40,984 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 16:53:40,987 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:40,989 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 16:53:40,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 16:53:41,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 16:53:41,009 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:41,009 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:41,009 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:41,009 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:41,009 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:41,012 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,39843,1676307218728, sessionid=0x1004e9a276c0000, setting cluster-up flag (Was=false) 2023-02-13 16:53:41,031 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:41,038 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 16:53:41,040 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:41,046 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:41,053 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 16:53:41,055 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:41,058 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.hbase-snapshot/.tmp 2023-02-13 16:53:41,142 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(952): ClusterId : 1a36896f-674b-48e6-9f6c-5070f3bef7ac 2023-02-13 16:53:41,142 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(952): ClusterId : 1a36896f-674b-48e6-9f6c-5070f3bef7ac 2023-02-13 16:53:41,142 INFO [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(952): ClusterId : 1a36896f-674b-48e6-9f6c-5070f3bef7ac 2023-02-13 16:53:41,149 DEBUG [RS:0;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:41,149 DEBUG [RS:1;jenkins-hbase4:37285] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:41,149 DEBUG [RS:2;jenkins-hbase4:45145] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:41,155 DEBUG [RS:2;jenkins-hbase4:45145] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:41,155 DEBUG [RS:1;jenkins-hbase4:37285] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:41,155 DEBUG [RS:0;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:41,155 DEBUG [RS:1;jenkins-hbase4:37285] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:41,155 DEBUG [RS:2;jenkins-hbase4:45145] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:41,155 DEBUG [RS:0;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:41,160 DEBUG [RS:1;jenkins-hbase4:37285] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:41,161 DEBUG [RS:2;jenkins-hbase4:45145] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:41,161 DEBUG [RS:0;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:41,163 DEBUG [RS:1;jenkins-hbase4:37285] zookeeper.ReadOnlyZKClient(139): Connect 0x14d131f2 to 127.0.0.1:59338 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:41,163 DEBUG [RS:2;jenkins-hbase4:45145] zookeeper.ReadOnlyZKClient(139): Connect 0x51d36f56 to 127.0.0.1:59338 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:41,163 DEBUG [RS:0;jenkins-hbase4:33173] zookeeper.ReadOnlyZKClient(139): Connect 0x623f8bdb to 127.0.0.1:59338 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:41,176 DEBUG [RS:0;jenkins-hbase4:33173] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6cf869ec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:41,177 DEBUG [RS:2;jenkins-hbase4:45145] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f148b1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:41,177 DEBUG [RS:1;jenkins-hbase4:37285] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a16dd18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:41,177 DEBUG [RS:2;jenkins-hbase4:45145] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b2b0426, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:41,177 DEBUG [RS:0;jenkins-hbase4:33173] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68c7e0ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:41,177 DEBUG [RS:1;jenkins-hbase4:37285] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e295a82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:41,185 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 16:53:41,195 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:41,195 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:41,195 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:41,195 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:41,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 16:53:41,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:41,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,202 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676307251202 2023-02-13 16:53:41,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 16:53:41,215 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:45145 2023-02-13 16:53:41,215 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:37285 2023-02-13 16:53:41,215 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 16:53:41,216 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 16:53:41,216 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:33173 2023-02-13 16:53:41,219 INFO [RS:1;jenkins-hbase4:37285] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:41,219 INFO [RS:0;jenkins-hbase4:33173] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:41,220 INFO [RS:0;jenkins-hbase4:33173] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:41,219 INFO [RS:2;jenkins-hbase4:45145] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:41,220 INFO [RS:2;jenkins-hbase4:45145] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:41,220 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:41,220 INFO [RS:1;jenkins-hbase4:37285] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:41,220 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:41,220 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:41,222 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:41,223 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 16:53:41,223 INFO [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39843,1676307218728 with isa=jenkins-hbase4.apache.org/172.31.14.131:45145, startcode=1676307220019 2023-02-13 16:53:41,224 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39843,1676307218728 with isa=jenkins-hbase4.apache.org/172.31.14.131:33173, startcode=1676307219927 2023-02-13 16:53:41,223 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39843,1676307218728 with isa=jenkins-hbase4.apache.org/172.31.14.131:37285, startcode=1676307219983 2023-02-13 16:53:41,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 16:53:41,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 16:53:41,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 16:53:41,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 16:53:41,233 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,234 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 16:53:41,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 16:53:41,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 16:53:41,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 16:53:41,240 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 16:53:41,241 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307221241,5,FailOnTimeoutGroup] 2023-02-13 16:53:41,242 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307221241,5,FailOnTimeoutGroup] 2023-02-13 16:53:41,242 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,242 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 16:53:41,243 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,243 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,243 DEBUG [RS:2;jenkins-hbase4:45145] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:41,243 DEBUG [RS:0;jenkins-hbase4:33173] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:41,243 DEBUG [RS:1;jenkins-hbase4:37285] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:41,273 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:41,274 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:41,275 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2 2023-02-13 16:53:41,309 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:41,315 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 16:53:41,318 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38443, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:41,318 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59387, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:41,318 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37509, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:41,321 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info 2023-02-13 16:53:41,322 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 16:53:41,324 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:41,324 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 16:53:41,327 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:41,327 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 16:53:41,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:41,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 16:53:41,331 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/table 2023-02-13 16:53:41,332 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 16:53:41,333 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39843] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:41,334 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:41,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39843] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:41,335 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:41,337 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740 2023-02-13 16:53:41,337 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740 2023-02-13 16:53:41,341 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 16:53:41,344 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 16:53:41,351 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:41,352 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60174681, jitterRate=-0.10332737863063812}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 16:53:41,352 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676307221310Initializing all the Stores at 1676307221312 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307221312Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307221315 (+3 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307221315Cleaning up temporary data from old regions at 1676307221339 (+24 ms)Cleaning up detritus from prior splits at 1676307221340 (+1 ms)Region opened successfully at 1676307221352 (+12 ms) 2023-02-13 16:53:41,353 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 16:53:41,353 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 16:53:41,353 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 16:53:41,353 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 16:53:41,353 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 16:53:41,354 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 16:53:41,354 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676307221353Disabling compacts and flushes for region at 1676307221353Disabling writes for close at 1676307221353Writing region close event to WAL at 1676307221354 (+1 ms)Closed at 1676307221354 2023-02-13 16:53:41,356 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2 2023-02-13 16:53:41,356 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2 2023-02-13 16:53:41,356 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2 2023-02-13 16:53:41,357 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39815 2023-02-13 16:53:41,357 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39815 2023-02-13 16:53:41,357 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:41,357 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39815 2023-02-13 16:53:41,357 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:41,358 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:41,365 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 16:53:41,365 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 16:53:41,367 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:41,369 DEBUG [RS:2;jenkins-hbase4:45145] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:41,369 WARN [RS:2;jenkins-hbase4:45145] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:41,369 DEBUG [RS:1;jenkins-hbase4:37285] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:41,369 DEBUG [RS:0;jenkins-hbase4:33173] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:41,371 WARN [RS:0;jenkins-hbase4:33173] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:41,369 INFO [RS:2;jenkins-hbase4:45145] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:41,371 INFO [RS:0;jenkins-hbase4:33173] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:41,372 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:41,372 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:41,371 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37285,1676307219983] 2023-02-13 16:53:41,370 WARN [RS:1;jenkins-hbase4:37285] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:41,377 INFO [RS:1;jenkins-hbase4:37285] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:41,375 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,45145,1676307220019] 2023-02-13 16:53:41,377 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33173,1676307219927] 2023-02-13 16:53:41,377 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:41,384 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 16:53:41,387 DEBUG [RS:0;jenkins-hbase4:33173] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:41,387 DEBUG [RS:1;jenkins-hbase4:37285] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:41,387 DEBUG [RS:2;jenkins-hbase4:45145] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:41,388 DEBUG [RS:1;jenkins-hbase4:37285] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:41,388 DEBUG [RS:2;jenkins-hbase4:45145] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:41,388 DEBUG [RS:0;jenkins-hbase4:33173] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:41,388 DEBUG [RS:1;jenkins-hbase4:37285] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:41,388 DEBUG [RS:2;jenkins-hbase4:45145] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:41,389 DEBUG [RS:0;jenkins-hbase4:33173] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:41,397 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 16:53:41,400 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:41,400 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:41,400 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:41,401 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 16:53:41,410 INFO [RS:0;jenkins-hbase4:33173] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:41,410 INFO [RS:2;jenkins-hbase4:45145] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:41,410 INFO [RS:1;jenkins-hbase4:37285] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:41,430 INFO [RS:1;jenkins-hbase4:37285] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:41,430 INFO [RS:0;jenkins-hbase4:33173] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:41,431 INFO [RS:2;jenkins-hbase4:45145] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:41,434 INFO [RS:1;jenkins-hbase4:37285] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:41,434 INFO [RS:2;jenkins-hbase4:45145] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:41,435 INFO [RS:1;jenkins-hbase4:37285] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,434 INFO [RS:0;jenkins-hbase4:33173] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:41,435 INFO [RS:2;jenkins-hbase4:45145] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,435 INFO [RS:0;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,436 INFO [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:41,436 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:41,436 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:41,444 INFO [RS:0;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,444 INFO [RS:1;jenkins-hbase4:37285] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,444 INFO [RS:2;jenkins-hbase4:45145] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,444 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,444 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,445 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:41,445 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:41,446 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,446 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,446 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,446 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,446 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,446 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,446 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,446 DEBUG [RS:0;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,446 DEBUG [RS:1;jenkins-hbase4:37285] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,446 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,447 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,447 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,447 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:41,448 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,448 INFO [RS:0;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,448 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,448 INFO [RS:0;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,448 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,448 INFO [RS:0;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,448 DEBUG [RS:2;jenkins-hbase4:45145] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:41,449 INFO [RS:1;jenkins-hbase4:37285] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,449 INFO [RS:1;jenkins-hbase4:37285] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,450 INFO [RS:1;jenkins-hbase4:37285] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,450 INFO [RS:2;jenkins-hbase4:45145] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,450 INFO [RS:2;jenkins-hbase4:45145] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,450 INFO [RS:2;jenkins-hbase4:45145] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,466 INFO [RS:0;jenkins-hbase4:33173] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:41,467 INFO [RS:2;jenkins-hbase4:45145] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:41,466 INFO [RS:1;jenkins-hbase4:37285] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:41,470 INFO [RS:0;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33173,1676307219927-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,470 INFO [RS:1;jenkins-hbase4:37285] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37285,1676307219983-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,470 INFO [RS:2;jenkins-hbase4:45145] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45145,1676307220019-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:41,491 INFO [RS:0;jenkins-hbase4:33173] regionserver.Replication(203): jenkins-hbase4.apache.org,33173,1676307219927 started 2023-02-13 16:53:41,491 INFO [RS:2;jenkins-hbase4:45145] regionserver.Replication(203): jenkins-hbase4.apache.org,45145,1676307220019 started 2023-02-13 16:53:41,491 INFO [RS:1;jenkins-hbase4:37285] regionserver.Replication(203): jenkins-hbase4.apache.org,37285,1676307219983 started 2023-02-13 16:53:41,491 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33173,1676307219927, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33173, sessionid=0x1004e9a276c0001 2023-02-13 16:53:41,492 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,37285,1676307219983, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37285, sessionid=0x1004e9a276c0002 2023-02-13 16:53:41,491 INFO [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,45145,1676307220019, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:45145, sessionid=0x1004e9a276c0003 2023-02-13 16:53:41,492 DEBUG [RS:0;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:41,492 DEBUG [RS:1;jenkins-hbase4:37285] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:41,492 DEBUG [RS:2;jenkins-hbase4:45145] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:41,492 DEBUG [RS:1;jenkins-hbase4:37285] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:41,492 DEBUG [RS:0;jenkins-hbase4:33173] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:41,493 DEBUG [RS:1;jenkins-hbase4:37285] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37285,1676307219983' 2023-02-13 16:53:41,493 DEBUG [RS:2;jenkins-hbase4:45145] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:41,493 DEBUG [RS:1;jenkins-hbase4:37285] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:41,493 DEBUG [RS:0;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33173,1676307219927' 2023-02-13 16:53:41,493 DEBUG [RS:2;jenkins-hbase4:45145] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45145,1676307220019' 2023-02-13 16:53:41,493 DEBUG [RS:2;jenkins-hbase4:45145] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:41,493 DEBUG [RS:0;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:41,494 DEBUG [RS:1;jenkins-hbase4:37285] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:41,494 DEBUG [RS:2;jenkins-hbase4:45145] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:41,494 DEBUG [RS:0;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:41,495 DEBUG [RS:1;jenkins-hbase4:37285] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:41,495 DEBUG [RS:1;jenkins-hbase4:37285] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:41,495 DEBUG [RS:2;jenkins-hbase4:45145] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:41,495 DEBUG [RS:0;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:41,495 DEBUG [RS:2;jenkins-hbase4:45145] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:41,495 DEBUG [RS:1;jenkins-hbase4:37285] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:41,495 DEBUG [RS:2;jenkins-hbase4:45145] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:41,495 DEBUG [RS:0;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:41,495 DEBUG [RS:2;jenkins-hbase4:45145] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45145,1676307220019' 2023-02-13 16:53:41,496 DEBUG [RS:2;jenkins-hbase4:45145] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:41,495 DEBUG [RS:1;jenkins-hbase4:37285] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37285,1676307219983' 2023-02-13 16:53:41,496 DEBUG [RS:0;jenkins-hbase4:33173] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:41,496 DEBUG [RS:0;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33173,1676307219927' 2023-02-13 16:53:41,496 DEBUG [RS:0;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:41,496 DEBUG [RS:1;jenkins-hbase4:37285] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:41,496 DEBUG [RS:2;jenkins-hbase4:45145] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:41,497 DEBUG [RS:0;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:41,497 DEBUG [RS:1;jenkins-hbase4:37285] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:41,497 DEBUG [RS:2;jenkins-hbase4:45145] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:41,497 DEBUG [RS:0;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:41,497 INFO [RS:2;jenkins-hbase4:45145] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:41,498 DEBUG [RS:1;jenkins-hbase4:37285] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:41,498 INFO [RS:2;jenkins-hbase4:45145] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:41,497 INFO [RS:0;jenkins-hbase4:33173] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:41,498 INFO [RS:1;jenkins-hbase4:37285] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:41,498 INFO [RS:0;jenkins-hbase4:33173] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:41,499 INFO [RS:1;jenkins-hbase4:37285] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:41,553 DEBUG [jenkins-hbase4:39843] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 16:53:41,557 DEBUG [jenkins-hbase4:39843] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:41,562 DEBUG [jenkins-hbase4:39843] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:41,563 DEBUG [jenkins-hbase4:39843] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:41,563 DEBUG [jenkins-hbase4:39843] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:41,563 DEBUG [jenkins-hbase4:39843] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:41,565 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33173,1676307219927, state=OPENING 2023-02-13 16:53:41,573 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 16:53:41,574 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:41,575 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 16:53:41,577 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33173,1676307219927}] 2023-02-13 16:53:41,609 INFO [RS:2;jenkins-hbase4:45145] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45145%2C1676307220019, suffix=, logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,45145,1676307220019, archiveDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs, maxLogs=32 2023-02-13 16:53:41,610 INFO [RS:0;jenkins-hbase4:33173] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33173%2C1676307219927, suffix=, logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927, archiveDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs, maxLogs=32 2023-02-13 16:53:41,609 INFO [RS:1;jenkins-hbase4:37285] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37285%2C1676307219983, suffix=, logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,37285,1676307219983, archiveDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs, maxLogs=32 2023-02-13 16:53:41,633 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK] 2023-02-13 16:53:41,633 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK] 2023-02-13 16:53:41,634 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK] 2023-02-13 16:53:41,642 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK] 2023-02-13 16:53:41,642 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK] 2023-02-13 16:53:41,642 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK] 2023-02-13 16:53:41,643 INFO [RS:2;jenkins-hbase4:45145] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,45145,1676307220019/jenkins-hbase4.apache.org%2C45145%2C1676307220019.1676307221614 2023-02-13 16:53:41,645 DEBUG [RS:2;jenkins-hbase4:45145] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK], DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK], DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK]] 2023-02-13 16:53:41,647 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK] 2023-02-13 16:53:41,647 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK] 2023-02-13 16:53:41,647 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK] 2023-02-13 16:53:41,654 INFO [RS:1;jenkins-hbase4:37285] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,37285,1676307219983/jenkins-hbase4.apache.org%2C37285%2C1676307219983.1676307221614 2023-02-13 16:53:41,654 INFO [RS:0;jenkins-hbase4:33173] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927/jenkins-hbase4.apache.org%2C33173%2C1676307219927.1676307221614 2023-02-13 16:53:41,654 DEBUG [RS:1;jenkins-hbase4:37285] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK], DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK], DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK]] 2023-02-13 16:53:41,656 DEBUG [RS:0;jenkins-hbase4:33173] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK], DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK], DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK]] 2023-02-13 16:53:41,767 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:41,770 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:41,780 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49012, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 16:53:41,795 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 16:53:41,795 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:41,799 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33173%2C1676307219927.meta, suffix=.meta, logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927, archiveDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs, maxLogs=32 2023-02-13 16:53:41,821 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK] 2023-02-13 16:53:41,822 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK] 2023-02-13 16:53:41,822 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK] 2023-02-13 16:53:41,830 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927/jenkins-hbase4.apache.org%2C33173%2C1676307219927.meta.1676307221801.meta 2023-02-13 16:53:41,830 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK], DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK], DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK]] 2023-02-13 16:53:41,830 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:41,832 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 16:53:41,849 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 16:53:41,854 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 16:53:41,859 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 16:53:41,859 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:41,859 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 16:53:41,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 16:53:41,863 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 16:53:41,865 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info 2023-02-13 16:53:41,865 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info 2023-02-13 16:53:41,865 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 16:53:41,866 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:41,867 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 16:53:41,869 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:41,869 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:41,869 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 16:53:41,870 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:41,871 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 16:53:41,872 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/table 2023-02-13 16:53:41,872 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/table 2023-02-13 16:53:41,873 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 16:53:41,873 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:41,875 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740 2023-02-13 16:53:41,881 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740 2023-02-13 16:53:41,886 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 16:53:41,888 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 16:53:41,890 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68083779, jitterRate=0.014527365565299988}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 16:53:41,890 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676307221860Writing region info on filesystem at 1676307221860Initializing all the Stores at 1676307221862 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307221862Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307221862Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307221862Cleaning up temporary data from old regions at 1676307221884 (+22 ms)Cleaning up detritus from prior splits at 1676307221885 (+1 ms)Running coprocessor post-open hooks at 1676307221890 (+5 ms)Region opened successfully at 1676307221890 2023-02-13 16:53:41,900 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676307221756 2023-02-13 16:53:41,917 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 16:53:41,918 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 16:53:41,919 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33173,1676307219927, state=OPEN 2023-02-13 16:53:41,923 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 16:53:41,923 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 16:53:41,927 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 16:53:41,928 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33173,1676307219927 in 346 msec 2023-02-13 16:53:41,933 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 16:53:41,933 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 545 msec 2023-02-13 16:53:41,939 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 835 msec 2023-02-13 16:53:41,939 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676307220064, completionTime=-1 2023-02-13 16:53:41,939 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 16:53:41,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 16:53:42,013 DEBUG [hconnection-0x12ca7048-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:42,017 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49026, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:53:42,038 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 16:53:42,038 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676307282038 2023-02-13 16:53:42,038 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676307342038 2023-02-13 16:53:42,038 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 98 msec 2023-02-13 16:53:42,063 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39843,1676307218728-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:42,063 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39843,1676307218728-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:42,063 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39843,1676307218728-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:42,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39843, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:42,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:42,083 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 16:53:42,085 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 16:53:42,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:42,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 16:53:42,101 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 16:53:42,105 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 16:53:42,128 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:42,130 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7 empty. 2023-02-13 16:53:42,131 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:42,131 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 16:53:42,176 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:42,178 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => e0ebffe3b7f1aa1f53e67e1b20582aa7, NAME => 'hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp 2023-02-13 16:53:42,199 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:42,199 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing e0ebffe3b7f1aa1f53e67e1b20582aa7, disabling compactions & flushes 2023-02-13 16:53:42,199 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:42,199 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:42,199 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. after waiting 0 ms 2023-02-13 16:53:42,199 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:42,199 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:42,199 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for e0ebffe3b7f1aa1f53e67e1b20582aa7: Waiting for close lock at 1676307222199Disabling compacts and flushes for region at 1676307222199Disabling writes for close at 1676307222199Writing region close event to WAL at 1676307222199Closed at 1676307222199 2023-02-13 16:53:42,204 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 16:53:42,218 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307222207"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307222207"}]},"ts":"1676307222207"} 2023-02-13 16:53:42,244 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 16:53:42,246 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 16:53:42,250 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307222246"}]},"ts":"1676307222246"} 2023-02-13 16:53:42,256 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 16:53:42,260 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:42,261 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:42,261 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:42,261 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:42,261 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:42,264 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e0ebffe3b7f1aa1f53e67e1b20582aa7, ASSIGN}] 2023-02-13 16:53:42,266 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e0ebffe3b7f1aa1f53e67e1b20582aa7, ASSIGN 2023-02-13 16:53:42,268 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e0ebffe3b7f1aa1f53e67e1b20582aa7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37285,1676307219983; forceNewPlan=false, retain=false 2023-02-13 16:53:42,420 INFO [jenkins-hbase4:39843] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:42,421 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e0ebffe3b7f1aa1f53e67e1b20582aa7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:42,421 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307222421"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307222421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307222421"}]},"ts":"1676307222421"} 2023-02-13 16:53:42,425 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e0ebffe3b7f1aa1f53e67e1b20582aa7, server=jenkins-hbase4.apache.org,37285,1676307219983}] 2023-02-13 16:53:42,579 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:42,580 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:42,584 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56334, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 16:53:42,590 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:42,592 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0ebffe3b7f1aa1f53e67e1b20582aa7, NAME => 'hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:42,593 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:42,593 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:42,593 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:42,593 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:42,595 INFO [StoreOpener-e0ebffe3b7f1aa1f53e67e1b20582aa7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:42,597 DEBUG [StoreOpener-e0ebffe3b7f1aa1f53e67e1b20582aa7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7/info 2023-02-13 16:53:42,597 DEBUG [StoreOpener-e0ebffe3b7f1aa1f53e67e1b20582aa7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7/info 2023-02-13 16:53:42,597 INFO [StoreOpener-e0ebffe3b7f1aa1f53e67e1b20582aa7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0ebffe3b7f1aa1f53e67e1b20582aa7 columnFamilyName info 2023-02-13 16:53:42,598 INFO [StoreOpener-e0ebffe3b7f1aa1f53e67e1b20582aa7-1] regionserver.HStore(310): Store=e0ebffe3b7f1aa1f53e67e1b20582aa7/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:42,600 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:42,601 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:42,605 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:42,609 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:42,610 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e0ebffe3b7f1aa1f53e67e1b20582aa7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68067880, jitterRate=0.014290452003479004}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:42,610 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e0ebffe3b7f1aa1f53e67e1b20582aa7: Running coprocessor pre-open hook at 1676307222593Writing region info on filesystem at 1676307222593Initializing all the Stores at 1676307222594 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307222594Cleaning up temporary data from old regions at 1676307222602 (+8 ms)Cleaning up detritus from prior splits at 1676307222602Running coprocessor post-open hooks at 1676307222610 (+8 ms)Region opened successfully at 1676307222610 2023-02-13 16:53:42,612 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7., pid=6, masterSystemTime=1676307222579 2023-02-13 16:53:42,616 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:42,617 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:42,618 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e0ebffe3b7f1aa1f53e67e1b20582aa7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:42,619 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307222618"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307222618"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307222618"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307222618"}]},"ts":"1676307222618"} 2023-02-13 16:53:42,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 16:53:42,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e0ebffe3b7f1aa1f53e67e1b20582aa7, server=jenkins-hbase4.apache.org,37285,1676307219983 in 198 msec 2023-02-13 16:53:42,631 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 16:53:42,632 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e0ebffe3b7f1aa1f53e67e1b20582aa7, ASSIGN in 363 msec 2023-02-13 16:53:42,633 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 16:53:42,633 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307222633"}]},"ts":"1676307222633"} 2023-02-13 16:53:42,636 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 16:53:42,639 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 16:53:42,642 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 552 msec 2023-02-13 16:53:42,702 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 16:53:42,703 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 16:53:42,703 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:42,725 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:42,729 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56342, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:53:42,742 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 16:53:42,765 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 16:53:42,771 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 37 msec 2023-02-13 16:53:42,776 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 16:53:42,789 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 16:53:42,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 19 msec 2023-02-13 16:53:42,807 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 16:53:42,810 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 16:53:42,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.733sec 2023-02-13 16:53:42,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 16:53:42,816 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 16:53:42,816 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 16:53:42,818 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39843,1676307218728-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 16:53:42,820 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39843,1676307218728-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 16:53:42,831 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 16:53:42,847 DEBUG [Listener at localhost/45541] zookeeper.ReadOnlyZKClient(139): Connect 0x6e8f77d7 to 127.0.0.1:59338 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:42,853 DEBUG [Listener at localhost/45541] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@209f8ec2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:42,869 DEBUG [hconnection-0x16c27a4b-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:42,881 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49034, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:53:42,893 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:42,900 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-13 16:53:42,900 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:42,901 INFO [Listener at localhost/45541] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-13 16:53:42,901 INFO [Listener at localhost/45541] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-13 16:53:42,912 DEBUG [Listener at localhost/45541] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 16:53:42,915 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47564, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 16:53:42,927 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:42,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-13 16:53:42,931 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 16:53:42,933 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 16:53:42,935 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-13 16:53:42,935 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableOne/e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:42,936 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableOne/e365cbf746e0b34d3a7f860529364362 empty. 2023-02-13 16:53:42,939 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableOne/e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:42,939 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-13 16:53:42,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 16:53:42,959 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:42,960 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => e365cbf746e0b34d3a7f860529364362, NAME => 'restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp 2023-02-13 16:53:42,977 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:42,977 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing e365cbf746e0b34d3a7f860529364362, disabling compactions & flushes 2023-02-13 16:53:42,977 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:42,977 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:42,977 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. after waiting 0 ms 2023-02-13 16:53:42,977 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:42,977 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:42,977 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for e365cbf746e0b34d3a7f860529364362: Waiting for close lock at 1676307222977Disabling compacts and flushes for region at 1676307222977Disabling writes for close at 1676307222977Writing region close event to WAL at 1676307222977Closed at 1676307222977 2023-02-13 16:53:42,981 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 16:53:42,983 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307222983"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307222983"}]},"ts":"1676307222983"} 2023-02-13 16:53:42,985 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 16:53:42,987 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 16:53:42,987 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307222987"}]},"ts":"1676307222987"} 2023-02-13 16:53:42,989 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-13 16:53:42,994 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:42,995 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:42,995 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:42,995 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:42,995 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:42,996 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=e365cbf746e0b34d3a7f860529364362, ASSIGN}] 2023-02-13 16:53:42,998 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=e365cbf746e0b34d3a7f860529364362, ASSIGN 2023-02-13 16:53:42,999 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=e365cbf746e0b34d3a7f860529364362, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45145,1676307220019; forceNewPlan=false, retain=false 2023-02-13 16:53:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 16:53:43,150 INFO [jenkins-hbase4:39843] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:43,151 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e365cbf746e0b34d3a7f860529364362, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:43,151 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307223151"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307223151"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307223151"}]},"ts":"1676307223151"} 2023-02-13 16:53:43,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure e365cbf746e0b34d3a7f860529364362, server=jenkins-hbase4.apache.org,45145,1676307220019}] 2023-02-13 16:53:43,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 16:53:43,308 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:43,308 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:43,310 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58606, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 16:53:43,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:43,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e365cbf746e0b34d3a7f860529364362, NAME => 'restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:43,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:43,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:43,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:43,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:43,320 INFO [StoreOpener-e365cbf746e0b34d3a7f860529364362-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:43,323 DEBUG [StoreOpener-e365cbf746e0b34d3a7f860529364362-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableOne/e365cbf746e0b34d3a7f860529364362/family 2023-02-13 16:53:43,323 DEBUG [StoreOpener-e365cbf746e0b34d3a7f860529364362-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableOne/e365cbf746e0b34d3a7f860529364362/family 2023-02-13 16:53:43,324 INFO [StoreOpener-e365cbf746e0b34d3a7f860529364362-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e365cbf746e0b34d3a7f860529364362 columnFamilyName family 2023-02-13 16:53:43,324 INFO [StoreOpener-e365cbf746e0b34d3a7f860529364362-1] regionserver.HStore(310): Store=e365cbf746e0b34d3a7f860529364362/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:43,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableOne/e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:43,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableOne/e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:43,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:43,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableOne/e365cbf746e0b34d3a7f860529364362/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:43,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e365cbf746e0b34d3a7f860529364362; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68644478, jitterRate=0.022882431745529175}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:43,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e365cbf746e0b34d3a7f860529364362: Running coprocessor pre-open hook at 1676307223318Writing region info on filesystem at 1676307223318Initializing all the Stores at 1676307223320 (+2 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307223320Cleaning up temporary data from old regions at 1676307223328 (+8 ms)Cleaning up detritus from prior splits at 1676307223329 (+1 ms)Running coprocessor post-open hooks at 1676307223336 (+7 ms)Region opened successfully at 1676307223336 2023-02-13 16:53:43,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362., pid=11, masterSystemTime=1676307223308 2023-02-13 16:53:43,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:43,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:43,346 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e365cbf746e0b34d3a7f860529364362, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:43,346 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307223345"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307223345"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307223345"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307223345"}]},"ts":"1676307223345"} 2023-02-13 16:53:43,351 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-13 16:53:43,352 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure e365cbf746e0b34d3a7f860529364362, server=jenkins-hbase4.apache.org,45145,1676307220019 in 194 msec 2023-02-13 16:53:43,355 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-13 16:53:43,355 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=e365cbf746e0b34d3a7f860529364362, ASSIGN in 357 msec 2023-02-13 16:53:43,356 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 16:53:43,357 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307223356"}]},"ts":"1676307223356"} 2023-02-13 16:53:43,358 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-13 16:53:43,361 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 16:53:43,364 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 434 msec 2023-02-13 16:53:43,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 16:53:43,552 INFO [Listener at localhost/45541] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-13 16:53:43,552 DEBUG [Listener at localhost/45541] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-13 16:53:43,554 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:43,558 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-13 16:53:43,558 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:43,559 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-13 16:53:43,561 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:43,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-13 16:53:43,564 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 16:53:43,564 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-13 16:53:43,565 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 16:53:43,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 16:53:43,568 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableTwo/9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:43,569 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableTwo/9cd1055f4447f4c2ce5480adf04b8a27 empty. 2023-02-13 16:53:43,569 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableTwo/9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:43,569 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-13 16:53:43,591 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:43,592 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => 9cd1055f4447f4c2ce5480adf04b8a27, NAME => 'restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp 2023-02-13 16:53:43,606 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:43,606 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing 9cd1055f4447f4c2ce5480adf04b8a27, disabling compactions & flushes 2023-02-13 16:53:43,606 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:43,606 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:43,606 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. after waiting 0 ms 2023-02-13 16:53:43,606 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:43,606 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:43,606 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for 9cd1055f4447f4c2ce5480adf04b8a27: Waiting for close lock at 1676307223606Disabling compacts and flushes for region at 1676307223606Disabling writes for close at 1676307223606Writing region close event to WAL at 1676307223606Closed at 1676307223606 2023-02-13 16:53:43,610 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 16:53:43,613 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307223612"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307223612"}]},"ts":"1676307223612"} 2023-02-13 16:53:43,615 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 16:53:43,616 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 16:53:43,616 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307223616"}]},"ts":"1676307223616"} 2023-02-13 16:53:43,619 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-13 16:53:43,626 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:43,628 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:43,628 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:43,628 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:43,628 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:43,628 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=9cd1055f4447f4c2ce5480adf04b8a27, ASSIGN}] 2023-02-13 16:53:43,630 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=9cd1055f4447f4c2ce5480adf04b8a27, ASSIGN 2023-02-13 16:53:43,632 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=9cd1055f4447f4c2ce5480adf04b8a27, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37285,1676307219983; forceNewPlan=false, retain=false 2023-02-13 16:53:43,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 16:53:43,782 INFO [jenkins-hbase4:39843] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:43,784 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=9cd1055f4447f4c2ce5480adf04b8a27, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:43,784 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307223783"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307223783"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307223783"}]},"ts":"1676307223783"} 2023-02-13 16:53:43,787 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 9cd1055f4447f4c2ce5480adf04b8a27, server=jenkins-hbase4.apache.org,37285,1676307219983}] 2023-02-13 16:53:43,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 16:53:43,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:43,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9cd1055f4447f4c2ce5480adf04b8a27, NAME => 'restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:43,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo 9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:43,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:43,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:43,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:43,949 INFO [StoreOpener-9cd1055f4447f4c2ce5480adf04b8a27-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:43,951 DEBUG [StoreOpener-9cd1055f4447f4c2ce5480adf04b8a27-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableTwo/9cd1055f4447f4c2ce5480adf04b8a27/family 2023-02-13 16:53:43,951 DEBUG [StoreOpener-9cd1055f4447f4c2ce5480adf04b8a27-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableTwo/9cd1055f4447f4c2ce5480adf04b8a27/family 2023-02-13 16:53:43,951 INFO [StoreOpener-9cd1055f4447f4c2ce5480adf04b8a27-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9cd1055f4447f4c2ce5480adf04b8a27 columnFamilyName family 2023-02-13 16:53:43,952 INFO [StoreOpener-9cd1055f4447f4c2ce5480adf04b8a27-1] regionserver.HStore(310): Store=9cd1055f4447f4c2ce5480adf04b8a27/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:43,954 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableTwo/9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:43,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableTwo/9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:43,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:43,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableTwo/9cd1055f4447f4c2ce5480adf04b8a27/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:43,962 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9cd1055f4447f4c2ce5480adf04b8a27; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66024184, jitterRate=-0.016162991523742676}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:43,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9cd1055f4447f4c2ce5480adf04b8a27: Running coprocessor pre-open hook at 1676307223947Writing region info on filesystem at 1676307223947Initializing all the Stores at 1676307223949 (+2 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307223949Cleaning up temporary data from old regions at 1676307223955 (+6 ms)Cleaning up detritus from prior splits at 1676307223956 (+1 ms)Running coprocessor post-open hooks at 1676307223962 (+6 ms)Region opened successfully at 1676307223962 2023-02-13 16:53:43,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27., pid=14, masterSystemTime=1676307223939 2023-02-13 16:53:43,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:43,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:43,967 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=9cd1055f4447f4c2ce5480adf04b8a27, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:43,968 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307223967"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307223967"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307223967"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307223967"}]},"ts":"1676307223967"} 2023-02-13 16:53:43,973 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-13 16:53:43,973 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 9cd1055f4447f4c2ce5480adf04b8a27, server=jenkins-hbase4.apache.org,37285,1676307219983 in 183 msec 2023-02-13 16:53:43,976 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-13 16:53:43,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=9cd1055f4447f4c2ce5480adf04b8a27, ASSIGN in 345 msec 2023-02-13 16:53:43,978 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 16:53:43,978 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307223978"}]},"ts":"1676307223978"} 2023-02-13 16:53:43,980 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-13 16:53:43,983 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 16:53:43,986 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 423 msec 2023-02-13 16:53:44,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 16:53:44,171 INFO [Listener at localhost/45541] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-13 16:53:44,171 DEBUG [Listener at localhost/45541] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-13 16:53:44,172 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:44,176 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-13 16:53:44,176 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:44,176 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-13 16:53:44,178 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:44,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-13 16:53:44,182 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 16:53:44,182 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-13 16:53:44,183 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 16:53:44,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 16:53:44,185 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableThree/705be376268d045667aeb77faed459bf 2023-02-13 16:53:44,186 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableThree/705be376268d045667aeb77faed459bf empty. 2023-02-13 16:53:44,187 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableThree/705be376268d045667aeb77faed459bf 2023-02-13 16:53:44,187 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-13 16:53:44,205 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:44,206 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => 705be376268d045667aeb77faed459bf, NAME => 'restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.tmp 2023-02-13 16:53:44,220 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:44,220 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing 705be376268d045667aeb77faed459bf, disabling compactions & flushes 2023-02-13 16:53:44,220 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:44,221 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:44,221 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. after waiting 0 ms 2023-02-13 16:53:44,221 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:44,221 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:44,221 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for 705be376268d045667aeb77faed459bf: Waiting for close lock at 1676307224220Disabling compacts and flushes for region at 1676307224220Disabling writes for close at 1676307224221 (+1 ms)Writing region close event to WAL at 1676307224221Closed at 1676307224221 2023-02-13 16:53:44,224 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 16:53:44,226 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307224226"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307224226"}]},"ts":"1676307224226"} 2023-02-13 16:53:44,228 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 16:53:44,230 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 16:53:44,230 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307224230"}]},"ts":"1676307224230"} 2023-02-13 16:53:44,232 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-13 16:53:44,236 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:44,237 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:44,237 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:44,237 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:44,237 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:44,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=705be376268d045667aeb77faed459bf, ASSIGN}] 2023-02-13 16:53:44,240 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=705be376268d045667aeb77faed459bf, ASSIGN 2023-02-13 16:53:44,241 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=705be376268d045667aeb77faed459bf, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33173,1676307219927; forceNewPlan=false, retain=false 2023-02-13 16:53:44,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 16:53:44,391 INFO [jenkins-hbase4:39843] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:44,392 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=705be376268d045667aeb77faed459bf, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:44,392 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307224392"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307224392"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307224392"}]},"ts":"1676307224392"} 2023-02-13 16:53:44,395 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 705be376268d045667aeb77faed459bf, server=jenkins-hbase4.apache.org,33173,1676307219927}] 2023-02-13 16:53:44,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 16:53:44,553 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:44,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 705be376268d045667aeb77faed459bf, NAME => 'restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:44,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 705be376268d045667aeb77faed459bf 2023-02-13 16:53:44,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:44,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 705be376268d045667aeb77faed459bf 2023-02-13 16:53:44,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 705be376268d045667aeb77faed459bf 2023-02-13 16:53:44,556 INFO [StoreOpener-705be376268d045667aeb77faed459bf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 705be376268d045667aeb77faed459bf 2023-02-13 16:53:44,558 DEBUG [StoreOpener-705be376268d045667aeb77faed459bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf/family 2023-02-13 16:53:44,558 DEBUG [StoreOpener-705be376268d045667aeb77faed459bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf/family 2023-02-13 16:53:44,558 INFO [StoreOpener-705be376268d045667aeb77faed459bf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 705be376268d045667aeb77faed459bf columnFamilyName family 2023-02-13 16:53:44,559 INFO [StoreOpener-705be376268d045667aeb77faed459bf-1] regionserver.HStore(310): Store=705be376268d045667aeb77faed459bf/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:44,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf 2023-02-13 16:53:44,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf 2023-02-13 16:53:44,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 705be376268d045667aeb77faed459bf 2023-02-13 16:53:44,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:44,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 705be376268d045667aeb77faed459bf; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66218058, jitterRate=-0.013274043798446655}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:44,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 705be376268d045667aeb77faed459bf: Running coprocessor pre-open hook at 1676307224554Writing region info on filesystem at 1676307224554Initializing all the Stores at 1676307224556 (+2 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307224556Cleaning up temporary data from old regions at 1676307224562 (+6 ms)Cleaning up detritus from prior splits at 1676307224563 (+1 ms)Running coprocessor post-open hooks at 1676307224573 (+10 ms)Region opened successfully at 1676307224573 2023-02-13 16:53:44,574 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf., pid=17, masterSystemTime=1676307224548 2023-02-13 16:53:44,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:44,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:44,577 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=705be376268d045667aeb77faed459bf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:44,578 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307224577"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307224577"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307224577"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307224577"}]},"ts":"1676307224577"} 2023-02-13 16:53:44,583 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-13 16:53:44,583 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 705be376268d045667aeb77faed459bf, server=jenkins-hbase4.apache.org,33173,1676307219927 in 185 msec 2023-02-13 16:53:44,586 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-13 16:53:44,586 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=705be376268d045667aeb77faed459bf, ASSIGN in 346 msec 2023-02-13 16:53:44,587 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 16:53:44,587 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307224587"}]},"ts":"1676307224587"} 2023-02-13 16:53:44,589 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-13 16:53:44,591 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 16:53:44,594 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 414 msec 2023-02-13 16:53:44,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39843] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 16:53:44,789 INFO [Listener at localhost/45541] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-13 16:53:44,789 DEBUG [Listener at localhost/45541] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-13 16:53:44,789 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:44,793 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-13 16:53:44,793 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:44,793 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-13 16:53:44,794 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:44,799 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:44,803 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:44,807 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:44,808 INFO [Listener at localhost/45541] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 16:53:44,816 INFO [Listener at localhost/45541] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 16:53:44,816 INFO [Listener at localhost/45541] master.TestRetainAssignmentOnRestart(157): Stopping HMaster and jenkins-hbase4.apache.org,33173,1676307219927 server 2023-02-13 16:53:44,816 INFO [Listener at localhost/45541] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase4:39843,5,FailOnTimeoutGroup] 2023-02-13 16:53:44,816 INFO [Listener at localhost/45541] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39843,1676307218728' ***** 2023-02-13 16:53:44,816 INFO [Listener at localhost/45541] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-13 16:53:44,817 DEBUG [M:0;jenkins-hbase4:39843] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cd60567, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:44,818 INFO [M:0;jenkins-hbase4:39843] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:44,819 DEBUG [M:0;jenkins-hbase4:39843] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x189a24d4 to 127.0.0.1:59338 2023-02-13 16:53:44,822 DEBUG [M:0;jenkins-hbase4:39843] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:44,823 INFO [M:0;jenkins-hbase4:39843] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39843,1676307218728; all regions closed. 2023-02-13 16:53:44,823 DEBUG [M:0;jenkins-hbase4:39843] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:44,823 DEBUG [M:0;jenkins-hbase4:39843] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 16:53:44,823 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 16:53:44,823 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307221241] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307221241,5,FailOnTimeoutGroup] 2023-02-13 16:53:44,825 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:44,825 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:44,826 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:44,826 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:44,826 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:44,825 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307221241] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307221241,5,FailOnTimeoutGroup] 2023-02-13 16:53:44,823 DEBUG [M:0;jenkins-hbase4:39843] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 16:53:44,827 INFO [M:0;jenkins-hbase4:39843] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 16:53:44,827 INFO [M:0;jenkins-hbase4:39843] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 16:53:44,827 INFO [Listener at localhost/45541] hbase.LocalHBaseCluster(360): Waiting on jenkins-hbase4.apache.org,39843,1676307218728 2023-02-13 16:53:44,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:44,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:44,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:44,828 INFO [M:0;jenkins-hbase4:39843] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 16:53:44,828 DEBUG [M:0;jenkins-hbase4:39843] master.HMaster(1502): Stopping service threads 2023-02-13 16:53:44,829 INFO [M:0;jenkins-hbase4:39843] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 16:53:44,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:44,829 INFO [M:0;jenkins-hbase4:39843] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 16:53:44,830 ERROR [M:0;jenkins-hbase4:39843] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] 2023-02-13 16:53:44,831 INFO [M:0;jenkins-hbase4:39843] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 16:53:44,831 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 16:53:44,831 DEBUG [M:0;jenkins-hbase4:39843] zookeeper.ZKUtil(398): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 16:53:44,832 WARN [M:0;jenkins-hbase4:39843] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 16:53:44,832 INFO [M:0;jenkins-hbase4:39843] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 16:53:44,832 INFO [M:0;jenkins-hbase4:39843] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 16:53:44,833 DEBUG [M:0;jenkins-hbase4:39843] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 16:53:44,833 INFO [M:0;jenkins-hbase4:39843] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:44,833 DEBUG [M:0;jenkins-hbase4:39843] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:44,833 DEBUG [M:0;jenkins-hbase4:39843] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 16:53:44,833 DEBUG [M:0;jenkins-hbase4:39843] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:44,834 INFO [M:0;jenkins-hbase4:39843] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.17 KB heapSize=76.14 KB 2023-02-13 16:53:44,901 INFO [M:0;jenkins-hbase4:39843] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.17 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/91aa9f0a254248219bc8913859cc519e 2023-02-13 16:53:44,941 DEBUG [M:0;jenkins-hbase4:39843] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/91aa9f0a254248219bc8913859cc519e as hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/91aa9f0a254248219bc8913859cc519e 2023-02-13 16:53:44,950 INFO [M:0;jenkins-hbase4:39843] regionserver.HStore(1080): Added hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/91aa9f0a254248219bc8913859cc519e, entries=17, sequenceid=141, filesize=8.5 K 2023-02-13 16:53:44,952 INFO [M:0;jenkins-hbase4:39843] regionserver.HRegion(2947): Finished flush of dataSize ~64.17 KB/65707, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 118ms, sequenceid=141, compaction requested=false 2023-02-13 16:53:44,956 INFO [M:0;jenkins-hbase4:39843] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:44,957 DEBUG [M:0;jenkins-hbase4:39843] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676307224832Disabling compacts and flushes for region at 1676307224832Disabling writes for close at 1676307224833 (+1 ms)Obtaining lock to block concurrent updates at 1676307224834 (+1 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676307224834Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65707, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676307224843 (+9 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676307224844 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676307224846 (+2 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676307224874 (+28 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676307224878 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676307224943 (+65 ms)Finished flush of dataSize ~64.17 KB/65707, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 118ms, sequenceid=141, compaction requested=false at 1676307224952 (+9 ms)Writing region close event to WAL at 1676307224956 (+4 ms)Closed at 1676307224956 2023-02-13 16:53:44,964 INFO [M:0;jenkins-hbase4:39843] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 16:53:44,964 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:44,965 INFO [M:0;jenkins-hbase4:39843] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39843 2023-02-13 16:53:44,974 DEBUG [M:0;jenkins-hbase4:39843] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39843,1676307218728 already deleted, retry=false 2023-02-13 16:53:45,045 WARN [RS-EventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39843 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:45,045 WARN [RS-EventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39843 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:45,046 DEBUG [RS-EventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39843 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 2023-02-13 16:53:45,046 DEBUG [RS-EventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39843 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 2023-02-13 16:53:45,047 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 16:53:45,047 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 16:53:45,059 WARN [RS-EventLoopGroup-5-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39843 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:45,059 DEBUG [RS-EventLoopGroup-5-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39843 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 2023-02-13 16:53:45,059 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 16:53:45,076 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:45,077 INFO [M:0;jenkins-hbase4:39843] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39843,1676307218728; zookeeper connection closed. 2023-02-13 16:53:45,077 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@65e3c819 rejected from java.util.concurrent.ThreadPoolExecutor@8e0b885[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:45,077 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:39843-0x1004e9a276c0000, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:45,077 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6515a9a9 rejected from java.util.concurrent.ThreadPoolExecutor@8e0b885[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:45,079 INFO [Listener at localhost/45541] hbase.MiniHBaseCluster(508): Stopping Thread[RS:0;jenkins-hbase4:33173,5,FailOnTimeoutGroup] 2023-02-13 16:53:45,079 INFO [Listener at localhost/45541] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33173,1676307219927' ***** 2023-02-13 16:53:45,079 INFO [Listener at localhost/45541] regionserver.HRegionServer(2310): STOPPED: Stopping rs 0 2023-02-13 16:53:45,079 INFO [Listener at localhost/45541] master.TestRetainAssignmentOnRestart(163): Sleeping a bit 2023-02-13 16:53:45,152 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:53:45,260 INFO [RS:0;jenkins-hbase4:33173] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:53:45,260 INFO [RS:0;jenkins-hbase4:33173] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:53:45,260 INFO [RS:0;jenkins-hbase4:33173] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:53:45,260 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(3304): Received CLOSE for 705be376268d045667aeb77faed459bf 2023-02-13 16:53:45,261 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:45,262 DEBUG [RS:0;jenkins-hbase4:33173] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x623f8bdb to 127.0.0.1:59338 2023-02-13 16:53:45,262 DEBUG [RS:0;jenkins-hbase4:33173] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:45,262 INFO [RS:0;jenkins-hbase4:33173] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:53:45,262 INFO [RS:0;jenkins-hbase4:33173] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:53:45,262 INFO [RS:0;jenkins-hbase4:33173] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:53:45,262 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 16:53:45,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 705be376268d045667aeb77faed459bf, disabling compactions & flushes 2023-02-13 16:53:45,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:45,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:45,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. after waiting 0 ms 2023-02-13 16:53:45,263 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 16:53:45,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:45,265 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 16:53:45,265 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 16:53:45,265 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 16:53:45,264 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1479): Online Regions={705be376268d045667aeb77faed459bf=restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 16:53:45,265 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 16:53:45,265 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 16:53:45,265 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=5.14 KB heapSize=9.42 KB 2023-02-13 16:53:45,267 DEBUG [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1505): Waiting on 1588230740, 705be376268d045667aeb77faed459bf 2023-02-13 16:53:45,276 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 16:53:45,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:45,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 705be376268d045667aeb77faed459bf: Waiting for close lock at 1676307225262Running coprocessor pre-close hooks at 1676307225262Disabling compacts and flushes for region at 1676307225262Disabling writes for close at 1676307225263 (+1 ms)Writing region close event to WAL at 1676307225265 (+2 ms)Running coprocessor post-close hooks at 1676307225278 (+13 ms)Closed at 1676307225278 2023-02-13 16:53:45,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:45,302 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.77 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/.tmp/info/bd9eed117c8442e6a7d4c53efc0f6799 2023-02-13 16:53:45,340 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/.tmp/table/c92a9e41f06a44d7a2c839c8db3fc7c4 2023-02-13 16:53:45,350 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/.tmp/info/bd9eed117c8442e6a7d4c53efc0f6799 as hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info/bd9eed117c8442e6a7d4c53efc0f6799 2023-02-13 16:53:45,357 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:45,360 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info/bd9eed117c8442e6a7d4c53efc0f6799, entries=40, sequenceid=24, filesize=9.5 K 2023-02-13 16:53:45,363 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/.tmp/table/c92a9e41f06a44d7a2c839c8db3fc7c4 as hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/table/c92a9e41f06a44d7a2c839c8db3fc7c4 2023-02-13 16:53:45,370 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/table/c92a9e41f06a44d7a2c839c8db3fc7c4, entries=8, sequenceid=24, filesize=5.0 K 2023-02-13 16:53:45,372 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 107ms, sequenceid=24, compaction requested=false 2023-02-13 16:53:45,374 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 16:53:45,385 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=1 2023-02-13 16:53:45,386 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 16:53:45,386 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 16:53:45,387 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676307225264Running coprocessor pre-close hooks at 1676307225264Disabling compacts and flushes for region at 1676307225264Disabling writes for close at 1676307225265 (+1 ms)Obtaining lock to block concurrent updates at 1676307225265Preparing flush snapshotting stores in 1588230740 at 1676307225265Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=5262, getHeapSize=9600, getOffHeapSize=0, getCellsCount=48 at 1676307225266 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676307225270 (+4 ms)Flushing 1588230740/info: creating writer at 1676307225270Flushing 1588230740/info: appending metadata at 1676307225277 (+7 ms)Flushing 1588230740/info: closing flushed file at 1676307225277Flushing 1588230740/table: creating writer at 1676307225320 (+43 ms)Flushing 1588230740/table: appending metadata at 1676307225324 (+4 ms)Flushing 1588230740/table: closing flushed file at 1676307225324Flushing 1588230740/info: reopening flushed file at 1676307225351 (+27 ms)Flushing 1588230740/table: reopening flushed file at 1676307225364 (+13 ms)Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 107ms, sequenceid=24, compaction requested=false at 1676307225372 (+8 ms)Writing region close event to WAL at 1676307225381 (+9 ms)Running coprocessor post-close hooks at 1676307225386 (+5 ms)Closed at 1676307225386 2023-02-13 16:53:45,387 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 16:53:45,467 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33173,1676307219927; all regions closed. 2023-02-13 16:53:45,477 DEBUG [RS:0;jenkins-hbase4:33173] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs 2023-02-13 16:53:45,477 INFO [RS:0;jenkins-hbase4:33173] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33173%2C1676307219927.meta:.meta(num 1676307221801) 2023-02-13 16:53:45,484 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 16:53:45,484 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 16:53:45,485 DEBUG [RS:0;jenkins-hbase4:33173] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs 2023-02-13 16:53:45,485 INFO [RS:0;jenkins-hbase4:33173] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33173%2C1676307219927:(num 1676307221614) 2023-02-13 16:53:45,485 DEBUG [RS:0;jenkins-hbase4:33173] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:45,485 INFO [RS:0;jenkins-hbase4:33173] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:45,485 INFO [RS:0;jenkins-hbase4:33173] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 16:53:45,486 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:45,486 INFO [RS:0;jenkins-hbase4:33173] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33173 2023-02-13 16:53:45,489 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:45,489 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:45,489 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:45,489 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:45,489 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:45,490 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@366e8f5 rejected from java.util.concurrent.ThreadPoolExecutor@7597c41[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:45,490 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:45,490 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@47467d2b rejected from java.util.concurrent.ThreadPoolExecutor@7597c41[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:45,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:45,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:45,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:45,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:45,493 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,33173,1676307219927 znode expired, triggering replicatorRemoved event 2023-02-13 16:53:45,493 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,33173,1676307219927 znode expired, triggering replicatorRemoved event 2023-02-13 16:53:45,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:45,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:45,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:45,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:45,591 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:45,591 INFO [RS:0;jenkins-hbase4:33173] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33173,1676307219927; zookeeper connection closed. 2023-02-13 16:53:45,591 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f24c539 rejected from java.util.concurrent.ThreadPoolExecutor@7597c41[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:45,592 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c0001, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:45,592 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@562f93c1 rejected from java.util.concurrent.ThreadPoolExecutor@7597c41[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:45,592 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@620b9c92] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@620b9c92 2023-02-13 16:53:46,051 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 16:53:46,051 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 16:53:47,054 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 16:53:47,054 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(2737): No master found; retry 2023-02-13 16:53:47,079 INFO [Listener at localhost/45541] master.TestRetainAssignmentOnRestart(166): Starting HMaster and region server jenkins-hbase4.apache.org,33173,1676307219927 second time with the same port 2023-02-13 16:53:47,092 INFO [Listener at localhost/45541] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:47,092 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:47,092 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:47,092 INFO [Listener at localhost/45541] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:47,092 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:47,092 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:47,093 INFO [Listener at localhost/45541] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:47,096 INFO [Listener at localhost/45541] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40475 2023-02-13 16:53:47,097 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:47,098 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:47,099 INFO [Listener at localhost/45541] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40475 connecting to ZooKeeper ensemble=127.0.0.1:59338 2023-02-13 16:53:47,104 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:404750x0, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:47,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:40475-0x1004e9a276c0009 connected 2023-02-13 16:53:47,108 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:47,108 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(162): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 16:53:47,109 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:47,112 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40475 2023-02-13 16:53:47,112 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40475 2023-02-13 16:53:47,116 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40475 2023-02-13 16:53:47,120 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40475 2023-02-13 16:53:47,120 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40475 2023-02-13 16:53:47,120 INFO [Listener at localhost/45541] master.HMaster(439): hbase.rootdir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2, hbase.cluster.distributed=false 2023-02-13 16:53:47,122 DEBUG [pool-218-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-13 16:53:47,123 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:47,124 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 16:53:47,125 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:47,127 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:47,127 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:47,127 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:47,127 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:47,128 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:47,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:47,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,40475,1676307227091 from backup master directory 2023-02-13 16:53:47,129 INFO [M:0;jenkins-hbase4:40475] regionserver.HRegionServer(952): ClusterId : 1a36896f-674b-48e6-9f6c-5070f3bef7ac 2023-02-13 16:53:47,131 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:47,131 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:47,131 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 16:53:47,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:47,141 INFO [Listener at localhost/45541] client.ConnectionUtils(127): regionserver/jenkins-hbase4:33173 server-side Connection retries=45 2023-02-13 16:53:47,141 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:47,141 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:47,141 INFO [Listener at localhost/45541] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:47,142 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:47,142 INFO [Listener at localhost/45541] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:47,142 INFO [Listener at localhost/45541] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:47,144 INFO [Listener at localhost/45541] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33173 2023-02-13 16:53:47,144 INFO [Listener at localhost/45541] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:47,147 DEBUG [Listener at localhost/45541] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:47,147 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:47,149 INFO [Listener at localhost/45541] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:47,151 INFO [Listener at localhost/45541] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33173 connecting to ZooKeeper ensemble=127.0.0.1:59338 2023-02-13 16:53:47,154 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:331730x0, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:47,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33173-0x1004e9a276c000a connected 2023-02-13 16:53:47,156 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:47,157 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 16:53:47,158 DEBUG [Listener at localhost/45541] zookeeper.ZKUtil(164): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:47,164 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33173 2023-02-13 16:53:47,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:47,165 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33173 2023-02-13 16:53:47,165 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33173 2023-02-13 16:53:47,172 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33173 2023-02-13 16:53:47,174 DEBUG [Listener at localhost/45541] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33173 2023-02-13 16:53:47,178 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(952): ClusterId : 1a36896f-674b-48e6-9f6c-5070f3bef7ac 2023-02-13 16:53:47,180 DEBUG [RS:3;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:47,183 DEBUG [RS:3;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:47,183 DEBUG [RS:3;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:47,185 DEBUG [RS:3;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:47,186 DEBUG [RS:3;jenkins-hbase4:33173] zookeeper.ReadOnlyZKClient(139): Connect 0x3ac28e5b to 127.0.0.1:59338 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:47,192 DEBUG [RS:3;jenkins-hbase4:33173] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34056773, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:47,192 DEBUG [RS:3;jenkins-hbase4:33173] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@303ae52b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:47,205 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase4:33173 2023-02-13 16:53:47,205 INFO [RS:3;jenkins-hbase4:33173] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:47,205 INFO [RS:3;jenkins-hbase4:33173] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:47,205 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:47,206 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40475,1676307227091 with isa=jenkins-hbase4.apache.org/172.31.14.131:33173, startcode=1676307227140 2023-02-13 16:53:47,206 DEBUG [RS:3;jenkins-hbase4:33173] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:47,214 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53125, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:47,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:47,217 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2eaf6f01 to 127.0.0.1:59338 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:47,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61a543ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:47,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:47,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 16:53:47,223 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:47,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,39843,1676307218728 to hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,39843,1676307218728-dead as it is dead 2023-02-13 16:53:47,233 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,39843,1676307218728-dead/jenkins-hbase4.apache.org%2C39843%2C1676307218728.1676307220526 2023-02-13 16:53:47,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,39843,1676307218728-dead/jenkins-hbase4.apache.org%2C39843%2C1676307218728.1676307220526 after 4ms 2023-02-13 16:53:47,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,39843,1676307218728-dead/jenkins-hbase4.apache.org%2C39843%2C1676307218728.1676307220526 to hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39843%2C1676307218728.1676307220526 2023-02-13 16:53:47,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,39843,1676307218728-dead 2023-02-13 16:53:47,240 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:47,241 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 16:53:47,241 WARN [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 16:53:47,243 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40475%2C1676307227091, suffix=, logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,40475,1676307227091, archiveDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/oldWALs, maxLogs=10 2023-02-13 16:53:47,259 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK] 2023-02-13 16:53:47,260 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK] 2023-02-13 16:53:47,260 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK] 2023-02-13 16:53:47,263 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/WALs/jenkins-hbase4.apache.org,40475,1676307227091/jenkins-hbase4.apache.org%2C40475%2C1676307227091.1676307227243 2023-02-13 16:53:47,265 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK], DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK], DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK]] 2023-02-13 16:53:47,266 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:47,266 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:47,266 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:47,266 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:47,268 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:47,270 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 16:53:47,270 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 16:53:47,278 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/91aa9f0a254248219bc8913859cc519e 2023-02-13 16:53:47,279 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:47,280 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-13 16:53:47,280 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39843%2C1676307218728.1676307220526 2023-02-13 16:53:47,297 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39843%2C1676307218728.1676307220526 2023-02-13 16:53:47,299 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39843%2C1676307218728.1676307220526 2023-02-13 16:53:47,303 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:47,306 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-13 16:53:47,307 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59370200, jitterRate=-0.11531507968902588}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:47,307 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676307227266Initializing all the Stores at 1676307227267 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307227267Cleaning up temporary data from old regions at 1676307227299 (+32 ms)Cleaning up detritus from prior splits at 1676307227300 (+1 ms)Region opened successfully at 1676307227307 (+7 ms) 2023-02-13 16:53:47,307 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 16:53:47,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 16:53:47,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 16:53:47,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 16:53:47,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 16:53:47,327 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-13 16:53:47,328 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-13 16:53:47,328 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-13 16:53:47,328 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-13 16:53:47,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-13 16:53:47,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-13 16:53:47,330 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-13 16:53:47,330 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 20 msec 2023-02-13 16:53:47,330 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 16:53:47,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-13 16:53:47,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,33173,1676307219927, table=hbase:meta, region=1588230740 2023-02-13 16:53:47,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-13 16:53:47,335 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:47,336 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:47,338 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33173,1676307219927 already deleted, retry=false 2023-02-13 16:53:47,338 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,33173,1676307219927 on jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:47,342 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40475,1676307227091 with isa=jenkins-hbase4.apache.org/172.31.14.131:33173, startcode=1676307227140 2023-02-13 16:53:47,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:47,344 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 16:53:47,344 WARN [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 200 ms and then retrying. 2023-02-13 16:53:47,348 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,33173,1676307219927, splitWal=true, meta=true 2023-02-13 16:53:47,348 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase4.apache.org,33173,1676307219927 (carryingMeta=true) jenkins-hbase4.apache.org,33173,1676307219927/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@24a4a9b7[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 16:53:47,356 DEBUG [RS:2;jenkins-hbase4:45145] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:47,356 DEBUG [RS:1;jenkins-hbase4:37285] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:47,359 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54977, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:47,359 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51879, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:47,360 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:47,360 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:545) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:47,362 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 16:53:47,362 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 16:53:47,364 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-13 16:53:47,365 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 16:53:47,366 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 16:53:47,367 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 16:53:47,368 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 16:53:47,369 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 16:53:47,369 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,40475,1676307227091, sessionid=0x1004e9a276c0009, setting cluster-up flag (Was=true) 2023-02-13 16:53:47,373 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 16:53:47,374 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:47,377 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 16:53:47,378 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:47,379 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/.hbase-snapshot/.tmp 2023-02-13 16:53:47,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:47,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:47,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:47,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:47,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 16:53:47,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:47,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676307257384 2023-02-13 16:53:47,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 16:53:47,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 16:53:47,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 16:53:47,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 16:53:47,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 16:53:47,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 16:53:47,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 16:53:47,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 16:53:47,390 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 16:53:47,390 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 16:53:47,390 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 16:53:47,390 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307227390,5,FailOnTimeoutGroup] 2023-02-13 16:53:47,390 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 16:53:47,391 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307227390,5,FailOnTimeoutGroup] 2023-02-13 16:53:47,391 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,391 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 16:53:47,391 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,391 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,391 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676307227125, completionTime=-1 2023-02-13 16:53:47,392 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 16:53:47,392 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33173,1676307219927; numProcessing=1 2023-02-13 16:53:47,393 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33173,1676307219927, splitWal=true, meta=true 2023-02-13 16:53:47,394 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33173,1676307219927, splitWal=true, meta=true, isMeta: true 2023-02-13 16:53:47,397 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927-splitting 2023-02-13 16:53:47,398 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927-splitting dir is empty, no logs to split. 2023-02-13 16:53:47,398 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,33173,1676307219927 WAL count=0, meta=true 2023-02-13 16:53:47,406 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927-splitting dir is empty, no logs to split. 2023-02-13 16:53:47,406 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,33173,1676307219927 WAL count=0, meta=true 2023-02-13 16:53:47,406 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,33173,1676307219927 WAL splitting is done? wals=0, meta=true 2023-02-13 16:53:47,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 16:53:47,410 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 16:53:47,412 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,33173,1676307219927; forceNewPlan=false, retain=false 2023-02-13 16:53:47,518 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 16:53:47,519 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'restartTableTwo' 2023-02-13 16:53:47,520 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'restartTableOne' 2023-02-13 16:53:47,545 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40475,1676307227091 with isa=jenkins-hbase4.apache.org/172.31.14.131:33173, startcode=1676307227140 2023-02-13 16:53:47,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,547 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2 2023-02-13 16:53:47,547 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39815 2023-02-13 16:53:47,548 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:47,549 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:47,549 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:47,549 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:47,550 DEBUG [RS:3;jenkins-hbase4:33173] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,550 WARN [RS:3;jenkins-hbase4:33173] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:47,550 INFO [RS:3;jenkins-hbase4:33173] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:47,550 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:47,550 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33173,1676307227140] 2023-02-13 16:53:47,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:47,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:47,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:47,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,555 DEBUG [RS:3;jenkins-hbase4:33173] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:47,555 DEBUG [RS:3;jenkins-hbase4:33173] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:47,555 DEBUG [RS:3;jenkins-hbase4:33173] zookeeper.ZKUtil(162): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,556 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:47,557 INFO [RS:3;jenkins-hbase4:33173] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:47,558 INFO [RS:3;jenkins-hbase4:33173] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:47,559 INFO [RS:3;jenkins-hbase4:33173] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:47,559 INFO [RS:3;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,559 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:47,560 INFO [RS:3;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,560 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:33173, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,560 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:33173, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,560 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:33173, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,560 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,560 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:33173, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,561 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:33173, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:47,561 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:33173, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,561 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:33173, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,561 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:33173, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,561 DEBUG [RS:3;jenkins-hbase4:33173] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:33173, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:47,561 INFO [RS:3;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,562 INFO [RS:3;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,562 INFO [RS:3;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,563 INFO [jenkins-hbase4:40475] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:47,564 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33173,1676307227140, state=OPENING 2023-02-13 16:53:47,569 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 16:53:47,569 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 16:53:47,569 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33173,1676307227140}] 2023-02-13 16:53:47,576 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=185ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 16:53:47,576 INFO [RS:3;jenkins-hbase4:33173] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:47,576 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPENING, ts=1676307227569, server=jenkins-hbase4.apache.org,33173,1676307227140}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 16:53:47,576 INFO [RS:3;jenkins-hbase4:33173] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33173,1676307227140-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:47,587 INFO [RS:3;jenkins-hbase4:33173] regionserver.Replication(203): jenkins-hbase4.apache.org,33173,1676307227140 started 2023-02-13 16:53:47,587 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33173,1676307227140, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33173, sessionid=0x1004e9a276c000a 2023-02-13 16:53:47,587 DEBUG [RS:3;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:47,587 DEBUG [RS:3;jenkins-hbase4:33173] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,587 DEBUG [RS:3;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33173,1676307227140' 2023-02-13 16:53:47,587 DEBUG [RS:3;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:47,587 DEBUG [RS:3;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:47,588 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:47,588 DEBUG [RS:3;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:47,588 DEBUG [RS:3;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:47,588 DEBUG [RS:3;jenkins-hbase4:33173] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,588 DEBUG [RS:3;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33173,1676307227140' 2023-02-13 16:53:47,588 DEBUG [RS:3;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:47,589 DEBUG [RS:3;jenkins-hbase4:33173] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:47,590 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34776, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:53:47,591 DEBUG [RS:3;jenkins-hbase4:33173] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:47,591 INFO [RS:3;jenkins-hbase4:33173] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:47,591 INFO [RS:3;jenkins-hbase4:33173] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:47,591 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33173] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:34776 deadline: 1676307287591, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,693 INFO [RS:3;jenkins-hbase4:33173] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33173%2C1676307227140, suffix=, logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307227140, archiveDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs, maxLogs=32 2023-02-13 16:53:47,695 WARN [ReadOnlyZKClient-127.0.0.1:59338@0x6e8f77d7] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-02-13 16:53:47,696 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33173] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:34776 deadline: 1676307287696, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,715 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK] 2023-02-13 16:53:47,715 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK] 2023-02-13 16:53:47,715 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK] 2023-02-13 16:53:47,717 INFO [RS:3;jenkins-hbase4:33173] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307227140/jenkins-hbase4.apache.org%2C33173%2C1676307227140.1676307227694 2023-02-13 16:53:47,718 DEBUG [RS:3;jenkins-hbase4:33173] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK], DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK], DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK]] 2023-02-13 16:53:47,725 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:47,725 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:47,727 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34788, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=AdminService 2023-02-13 16:53:47,731 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 16:53:47,732 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:47,733 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33173%2C1676307227140.meta, suffix=.meta, logDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307227140, archiveDir=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs, maxLogs=32 2023-02-13 16:53:47,751 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK] 2023-02-13 16:53:47,751 DEBUG [RS-EventLoopGroup-8-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK] 2023-02-13 16:53:47,752 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK] 2023-02-13 16:53:47,756 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307227140/jenkins-hbase4.apache.org%2C33173%2C1676307227140.meta.1676307227734.meta 2023-02-13 16:53:47,756 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44649,DS-34034e11-090f-4563-b231-61dfc67b7a08,DISK], DatanodeInfoWithStorage[127.0.0.1:34747,DS-42509378-a03d-45f3-848b-61755295c521,DISK], DatanodeInfoWithStorage[127.0.0.1:41403,DS-c1c2bee4-44d5-4f11-ae2e-92004a004a9a,DISK]] 2023-02-13 16:53:47,756 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:47,757 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 16:53:47,757 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 16:53:47,758 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 16:53:47,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 16:53:47,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:47,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 16:53:47,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 16:53:47,759 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 16:53:47,760 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info 2023-02-13 16:53:47,761 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info 2023-02-13 16:53:47,761 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 16:53:47,769 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info/bd9eed117c8442e6a7d4c53efc0f6799 2023-02-13 16:53:47,769 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:47,769 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 16:53:47,770 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:47,770 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:47,770 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 16:53:47,771 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:47,771 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 16:53:47,772 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/table 2023-02-13 16:53:47,772 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/table 2023-02-13 16:53:47,773 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 16:53:47,780 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/table/c92a9e41f06a44d7a2c839c8db3fc7c4 2023-02-13 16:53:47,780 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:47,781 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740 2023-02-13 16:53:47,784 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740 2023-02-13 16:53:47,786 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 16:53:47,788 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 16:53:47,789 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=28; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66761715, jitterRate=-0.00517292320728302}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 16:53:47,789 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676307227758Writing region info on filesystem at 1676307227758Initializing all the Stores at 1676307227759 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307227759Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307227759Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307227759Cleaning up temporary data from old regions at 1676307227785 (+26 ms)Cleaning up detritus from prior splits at 1676307227785Running coprocessor post-open hooks at 1676307227789 (+4 ms)Region opened successfully at 1676307227789 2023-02-13 16:53:47,790 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=20, masterSystemTime=1676307227725 2023-02-13 16:53:47,794 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 16:53:47,794 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:33173-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 16:53:47,795 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33173,1676307227140, state=OPEN 2023-02-13 16:53:47,797 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 16:53:47,797 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 16:53:47,800 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-02-13 16:53:47,800 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33173,1676307227140 in 228 msec 2023-02-13 16:53:47,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-02-13 16:53:47,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 392 msec 2023-02-13 16:53:47,913 WARN [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39843 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:47,914 WARN [Listener at localhost/45541] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:39843 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:47,914 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39843 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39843 2023-02-13 16:53:47,914 DEBUG [Listener at localhost/45541] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39843 this server is in the failed servers list 2023-02-13 16:53:47,915 WARN [Listener at localhost/45541] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:39843 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:39843 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnSingleRSRestart(TestRetainAssignmentOnRestart.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:39843 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-13 16:53:47,916 DEBUG [Listener at localhost/45541] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 16:53:47,918 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47780, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 16:53:47,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:47,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.CallRunner(144): callId: 50 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:47780 deadline: 1676307287922, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 16:53:48,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:48,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.CallRunner(144): callId: 52 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:47780 deadline: 1676307288024, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 16:53:48,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:48,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.CallRunner(144): callId: 54 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:47780 deadline: 1676307288226, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 16:53:48,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:48,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40475] ipc.CallRunner(144): callId: 56 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:47780 deadline: 1676307288529, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 16:53:48,576 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 16:53:48,580 DEBUG [hconnection-0x41d58bb0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:48,581 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34790, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=ClientService 2023-02-13 16:53:48,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=e0ebffe3b7f1aa1f53e67e1b20582aa7, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,37285,1676307219983, regionLocation=jenkins-hbase4.apache.org,37285,1676307219983, openSeqNum=2 2023-02-13 16:53:48,585 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=e365cbf746e0b34d3a7f860529364362, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,45145,1676307220019, regionLocation=jenkins-hbase4.apache.org,45145,1676307220019, openSeqNum=2 2023-02-13 16:53:48,585 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=705be376268d045667aeb77faed459bf, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,33173,1676307219927, regionLocation=jenkins-hbase4.apache.org,33173,1676307219927, openSeqNum=2 2023-02-13 16:53:48,585 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=9cd1055f4447f4c2ce5480adf04b8a27, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,37285,1676307219983, regionLocation=jenkins-hbase4.apache.org,37285,1676307219983, openSeqNum=2 2023-02-13 16:53:48,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 16:53:48,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676307288585 2023-02-13 16:53:48,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676307348585 2023-02-13 16:53:48,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 8 msec 2023-02-13 16:53:48,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40475,1676307227091-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:48,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40475,1676307227091-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:48,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40475,1676307227091-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:48,612 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,33173,1676307219927 had 2 regions 2023-02-13 16:53:48,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:40475, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:48,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:48,613 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-13 16:53:48,616 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33173,1676307219927, splitWal=true, meta=true, isMeta: false 2023-02-13 16:53:48,619 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:48,619 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927-splitting dir is empty, no logs to split. 2023-02-13 16:53:48,619 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,33173,1676307219927 WAL count=0, meta=false 2023-02-13 16:53:48,620 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51350, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=ClientService 2023-02-13 16:53:48,621 WARN [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase4.apache.org,33173,1676307219927/restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:48,622 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307219927-splitting dir is empty, no logs to split. 2023-02-13 16:53:48,622 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,33173,1676307219927 WAL count=0, meta=false 2023-02-13 16:53:48,622 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,33173,1676307219927 WAL splitting is done? wals=0, meta=false 2023-02-13 16:53:48,625 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=705be376268d045667aeb77faed459bf, ASSIGN}] 2023-02-13 16:53:48,627 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=705be376268d045667aeb77faed459bf, ASSIGN 2023-02-13 16:53:48,628 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33173,1676307219927 2023-02-13 16:53:48,628 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=705be376268d045667aeb77faed459bf, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,33173,1676307219927; forceNewPlan=false, retain=true 2023-02-13 16:53:48,630 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 16:53:48,634 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 16:53:48,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 1.503sec 2023-02-13 16:53:48,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 16:53:48,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 16:53:48,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 16:53:48,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40475,1676307227091-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 16:53:48,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40475,1676307227091-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 16:53:48,639 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 16:53:48,778 INFO [jenkins-hbase4:40475] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:48,779 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=705be376268d045667aeb77faed459bf, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:48,779 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307228779"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307228779"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307228779"}]},"ts":"1676307228779"} 2023-02-13 16:53:48,782 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; OpenRegionProcedure 705be376268d045667aeb77faed459bf, server=jenkins-hbase4.apache.org,33173,1676307227140}] 2023-02-13 16:53:48,883 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 16:53:48,884 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-13 16:53:48,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:48,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(7850): Opening region: {ENCODED => 705be376268d045667aeb77faed459bf, NAME => 'restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:48,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 705be376268d045667aeb77faed459bf 2023-02-13 16:53:48,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:48,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(7890): checking encryption for 705be376268d045667aeb77faed459bf 2023-02-13 16:53:48,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(7893): checking classloading for 705be376268d045667aeb77faed459bf 2023-02-13 16:53:48,942 INFO [StoreOpener-705be376268d045667aeb77faed459bf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 705be376268d045667aeb77faed459bf 2023-02-13 16:53:48,943 DEBUG [StoreOpener-705be376268d045667aeb77faed459bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf/family 2023-02-13 16:53:48,943 DEBUG [StoreOpener-705be376268d045667aeb77faed459bf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf/family 2023-02-13 16:53:48,944 INFO [StoreOpener-705be376268d045667aeb77faed459bf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 705be376268d045667aeb77faed459bf columnFamilyName family 2023-02-13 16:53:48,945 INFO [StoreOpener-705be376268d045667aeb77faed459bf-1] regionserver.HStore(310): Store=705be376268d045667aeb77faed459bf/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:48,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf 2023-02-13 16:53:48,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf 2023-02-13 16:53:48,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1054): writing seq id for 705be376268d045667aeb77faed459bf 2023-02-13 16:53:48,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1071): Opened 705be376268d045667aeb77faed459bf; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64345322, jitterRate=-0.04117998480796814}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:48,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(964): Region open journal for 705be376268d045667aeb77faed459bf: Running coprocessor pre-open hook at 1676307228940Writing region info on filesystem at 1676307228940Initializing all the Stores at 1676307228941 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307228941Cleaning up temporary data from old regions at 1676307228948 (+7 ms)Cleaning up detritus from prior splits at 1676307228949 (+1 ms)Running coprocessor post-open hooks at 1676307228952 (+3 ms)Region opened successfully at 1676307228952 2023-02-13 16:53:48,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf., pid=22, masterSystemTime=1676307228935 2023-02-13 16:53:48,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:48,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:33173-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:48,956 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=705be376268d045667aeb77faed459bf, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:48,957 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307228956"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307228956"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307228956"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307228956"}]},"ts":"1676307228956"} 2023-02-13 16:53:48,962 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-02-13 16:53:48,962 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; OpenRegionProcedure 705be376268d045667aeb77faed459bf, server=jenkins-hbase4.apache.org,33173,1676307227140 in 177 msec 2023-02-13 16:53:48,966 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=18 2023-02-13 16:53:48,967 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,33173,1676307219927 after splitting done 2023-02-13 16:53:48,967 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=705be376268d045667aeb77faed459bf, ASSIGN in 337 msec 2023-02-13 16:53:48,967 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase4.apache.org,33173,1676307219927 from processing; numProcessing=0 2023-02-13 16:53:48,970 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,33173,1676307219927, splitWal=true, meta=true in 1.6280 sec 2023-02-13 16:53:49,040 DEBUG [hconnection-0x16c27a4b-shared-pool-6] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:49,042 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50896, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:53:49,046 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:49,055 DEBUG [hconnection-0x16c27a4b-shared-pool-9] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:49,066 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51356, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:53:49,069 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:49,084 INFO [Listener at localhost/45541] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:49,084 INFO [Listener at localhost/45541] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 16:53:49,088 INFO [Listener at localhost/45541] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 16:53:49,088 INFO [Listener at localhost/45541] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 705be376268d045667aeb77faed459bf, NAME => 'restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,33173,1676307219927, currentServer=jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:49,089 INFO [Listener at localhost/45541] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => e365cbf746e0b34d3a7f860529364362, NAME => 'restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,45145,1676307220019, currentServer=jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:49,089 INFO [Listener at localhost/45541] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => e0ebffe3b7f1aa1f53e67e1b20582aa7, NAME => 'hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,37285,1676307219983, currentServer=jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:49,089 INFO [Listener at localhost/45541] master.TestRetainAssignmentOnRestart(187): Key={ENCODED => 9cd1055f4447f4c2ce5480adf04b8a27, NAME => 'restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,37285,1676307219983, currentServer=jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:49,089 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 16:53:49,089 INFO [Listener at localhost/45541] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 16:53:49,090 DEBUG [Listener at localhost/45541] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e8f77d7 to 127.0.0.1:59338 2023-02-13 16:53:49,090 DEBUG [Listener at localhost/45541] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:49,091 DEBUG [Listener at localhost/45541] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 16:53:49,091 DEBUG [Listener at localhost/45541] util.JVMClusterUtil(257): Found active master hash=693033476, stopped=false 2023-02-13 16:53:49,091 INFO [Listener at localhost/45541] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:49,094 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:49,094 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:49,094 INFO [Listener at localhost/45541] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 16:53:49,094 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:49,094 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:49,096 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:49,098 DEBUG [Listener at localhost/45541] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2eaf6f01 to 127.0.0.1:59338 2023-02-13 16:53:49,098 DEBUG [Listener at localhost/45541] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:49,098 INFO [Listener at localhost/45541] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37285,1676307219983' ***** 2023-02-13 16:53:49,098 INFO [Listener at localhost/45541] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 16:53:49,098 INFO [Listener at localhost/45541] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,45145,1676307220019' ***** 2023-02-13 16:53:49,099 INFO [Listener at localhost/45541] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 16:53:49,099 INFO [RS:1;jenkins-hbase4:37285] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:53:49,099 INFO [RS:2;jenkins-hbase4:45145] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:53:49,099 INFO [Listener at localhost/45541] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33173,1676307227140' ***** 2023-02-13 16:53:49,099 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:53:49,099 INFO [Listener at localhost/45541] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 16:53:49,099 INFO [RS:2;jenkins-hbase4:45145] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:53:49,100 INFO [RS:3;jenkins-hbase4:33173] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:53:49,099 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:53:49,100 INFO [RS:3;jenkins-hbase4:33173] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:53:49,099 INFO [RS:1;jenkins-hbase4:37285] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:53:49,101 INFO [RS:3;jenkins-hbase4:33173] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:53:49,100 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:53:49,101 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(3304): Received CLOSE for 705be376268d045667aeb77faed459bf 2023-02-13 16:53:49,100 INFO [RS:2;jenkins-hbase4:45145] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:53:49,101 INFO [RS:1;jenkins-hbase4:37285] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:53:49,102 INFO [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(3304): Received CLOSE for e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:49,102 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:49,102 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(3304): Received CLOSE for 9cd1055f4447f4c2ce5480adf04b8a27 2023-02-13 16:53:49,102 DEBUG [RS:3;jenkins-hbase4:33173] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ac28e5b to 127.0.0.1:59338 2023-02-13 16:53:49,102 DEBUG [RS:3;jenkins-hbase4:33173] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:49,102 INFO [RS:3;jenkins-hbase4:33173] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:53:49,102 INFO [RS:3;jenkins-hbase4:33173] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:53:49,102 INFO [RS:3;jenkins-hbase4:33173] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:53:49,102 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 16:53:49,102 INFO [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:49,102 DEBUG [RS:2;jenkins-hbase4:45145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51d36f56 to 127.0.0.1:59338 2023-02-13 16:53:49,103 DEBUG [RS:2;jenkins-hbase4:45145] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:49,103 INFO [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 16:53:49,103 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1479): Online Regions={e365cbf746e0b34d3a7f860529364362=restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362.} 2023-02-13 16:53:49,103 DEBUG [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1505): Waiting on e365cbf746e0b34d3a7f860529364362 2023-02-13 16:53:49,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1603): Closing 705be376268d045667aeb77faed459bf, disabling compactions & flushes 2023-02-13 16:53:49,104 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(3304): Received CLOSE for e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:49,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e365cbf746e0b34d3a7f860529364362, disabling compactions & flushes 2023-02-13 16:53:49,105 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 16:53:49,105 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:49,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9cd1055f4447f4c2ce5480adf04b8a27, disabling compactions & flushes 2023-02-13 16:53:49,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:49,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:49,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:49,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. after waiting 0 ms 2023-02-13 16:53:49,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:49,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. after waiting 0 ms 2023-02-13 16:53:49,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:49,106 DEBUG [RS:1;jenkins-hbase4:37285] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14d131f2 to 127.0.0.1:59338 2023-02-13 16:53:49,108 DEBUG [RS:1;jenkins-hbase4:37285] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:49,106 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 16:53:49,105 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1479): Online Regions={705be376268d045667aeb77faed459bf=restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf., 1588230740=hbase:meta,,1.1588230740} 2023-02-13 16:53:49,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:49,108 DEBUG [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1505): Waiting on 1588230740, 705be376268d045667aeb77faed459bf 2023-02-13 16:53:49,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:49,108 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 16:53:49,108 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1479): Online Regions={9cd1055f4447f4c2ce5480adf04b8a27=restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27., e0ebffe3b7f1aa1f53e67e1b20582aa7=hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7.} 2023-02-13 16:53:49,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:49,108 DEBUG [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1505): Waiting on 9cd1055f4447f4c2ce5480adf04b8a27, e0ebffe3b7f1aa1f53e67e1b20582aa7 2023-02-13 16:53:49,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. after waiting 0 ms 2023-02-13 16:53:49,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:49,108 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 16:53:49,109 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 16:53:49,109 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 16:53:49,109 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 16:53:49,109 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=994 B heapSize=2.31 KB 2023-02-13 16:53:49,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableOne/e365cbf746e0b34d3a7f860529364362/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 16:53:49,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:49,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableThree/705be376268d045667aeb77faed459bf/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 16:53:49,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e365cbf746e0b34d3a7f860529364362: Waiting for close lock at 1676307229102Running coprocessor pre-close hooks at 1676307229102Disabling compacts and flushes for region at 1676307229102Disabling writes for close at 1676307229109 (+7 ms)Writing region close event to WAL at 1676307229110 (+1 ms)Running coprocessor post-close hooks at 1676307229129 (+19 ms)Closed at 1676307229130 (+1 ms) 2023-02-13 16:53:49,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/default/restartTableTwo/9cd1055f4447f4c2ce5480adf04b8a27/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 16:53:49,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676307222923.e365cbf746e0b34d3a7f860529364362. 2023-02-13 16:53:49,133 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1837): Closed restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:49,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1557): Region close journal for 705be376268d045667aeb77faed459bf: Waiting for close lock at 1676307229102Running coprocessor pre-close hooks at 1676307229102Disabling compacts and flushes for region at 1676307229102Disabling writes for close at 1676307229108 (+6 ms)Writing region close event to WAL at 1676307229110 (+2 ms)Running coprocessor post-close hooks at 1676307229133 (+23 ms)Closed at 1676307229133 2023-02-13 16:53:49,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:33173-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676307224178.705be376268d045667aeb77faed459bf. 2023-02-13 16:53:49,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:49,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9cd1055f4447f4c2ce5480adf04b8a27: Waiting for close lock at 1676307229105Running coprocessor pre-close hooks at 1676307229105Disabling compacts and flushes for region at 1676307229105Disabling writes for close at 1676307229106 (+1 ms)Writing region close event to WAL at 1676307229114 (+8 ms)Running coprocessor post-close hooks at 1676307229134 (+20 ms)Closed at 1676307229134 2023-02-13 16:53:49,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676307223560.9cd1055f4447f4c2ce5480adf04b8a27. 2023-02-13 16:53:49,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e0ebffe3b7f1aa1f53e67e1b20582aa7, disabling compactions & flushes 2023-02-13 16:53:49,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:49,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:49,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. after waiting 0 ms 2023-02-13 16:53:49,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:49,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e0ebffe3b7f1aa1f53e67e1b20582aa7 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 16:53:49,156 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=994 B at sequenceid=32 (bloomFilter=false), to=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/.tmp/info/082583c2f96f4f9aaf41931dd658d93b 2023-02-13 16:53:49,163 INFO [regionserver/jenkins-hbase4:33173.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:49,171 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/.tmp/info/082583c2f96f4f9aaf41931dd658d93b as hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info/082583c2f96f4f9aaf41931dd658d93b 2023-02-13 16:53:49,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7/.tmp/info/6269eccecb1042a18ca8c702c2b7031f 2023-02-13 16:53:49,176 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:49,177 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:49,180 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HStore(1080): Added hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/info/082583c2f96f4f9aaf41931dd658d93b, entries=8, sequenceid=32, filesize=5.7 K 2023-02-13 16:53:49,181 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(2947): Finished flush of dataSize ~994 B/994, heapSize ~1.80 KB/1840, currentSize=0 B/0 for 1588230740 in 72ms, sequenceid=32, compaction requested=false 2023-02-13 16:53:49,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7/.tmp/info/6269eccecb1042a18ca8c702c2b7031f as hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7/info/6269eccecb1042a18ca8c702c2b7031f 2023-02-13 16:53:49,191 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/meta/1588230740/recovered.edits/35.seqid, newMaxSeqId=35, maxSeqId=27 2023-02-13 16:53:49,191 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 16:53:49,192 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 16:53:49,192 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676307229105Running coprocessor pre-close hooks at 1676307229105Disabling compacts and flushes for region at 1676307229105Disabling writes for close at 1676307229109 (+4 ms)Obtaining lock to block concurrent updates at 1676307229109Preparing flush snapshotting stores in 1588230740 at 1676307229109Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=994, getHeapSize=2320, getOffHeapSize=0, getCellsCount=8 at 1676307229110 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676307229110Flushing 1588230740/info: creating writer at 1676307229111 (+1 ms)Flushing 1588230740/info: appending metadata at 1676307229129 (+18 ms)Flushing 1588230740/info: closing flushed file at 1676307229129Flushing 1588230740/info: reopening flushed file at 1676307229172 (+43 ms)Finished flush of dataSize ~994 B/994, heapSize ~1.80 KB/1840, currentSize=0 B/0 for 1588230740 in 72ms, sequenceid=32, compaction requested=false at 1676307229181 (+9 ms)Writing region close event to WAL at 1676307229186 (+5 ms)Running coprocessor post-close hooks at 1676307229191 (+5 ms)Closed at 1676307229192 (+1 ms) 2023-02-13 16:53:49,192 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:33173-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 16:53:49,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7/info/6269eccecb1042a18ca8c702c2b7031f, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 16:53:49,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e0ebffe3b7f1aa1f53e67e1b20582aa7 in 60ms, sequenceid=6, compaction requested=false 2023-02-13 16:53:49,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:49,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:49,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:49,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:49,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/data/hbase/namespace/e0ebffe3b7f1aa1f53e67e1b20582aa7/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 16:53:49,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:49,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e0ebffe3b7f1aa1f53e67e1b20582aa7: Waiting for close lock at 1676307229135Running coprocessor pre-close hooks at 1676307229135Disabling compacts and flushes for region at 1676307229135Disabling writes for close at 1676307229135Obtaining lock to block concurrent updates at 1676307229135Preparing flush snapshotting stores in e0ebffe3b7f1aa1f53e67e1b20582aa7 at 1676307229135Finished memstore snapshotting hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676307229136 (+1 ms)Flushing stores of hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. at 1676307229137 (+1 ms)Flushing e0ebffe3b7f1aa1f53e67e1b20582aa7/info: creating writer at 1676307229137Flushing e0ebffe3b7f1aa1f53e67e1b20582aa7/info: appending metadata at 1676307229153 (+16 ms)Flushing e0ebffe3b7f1aa1f53e67e1b20582aa7/info: closing flushed file at 1676307229153Flushing e0ebffe3b7f1aa1f53e67e1b20582aa7/info: reopening flushed file at 1676307229187 (+34 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e0ebffe3b7f1aa1f53e67e1b20582aa7 in 60ms, sequenceid=6, compaction requested=false at 1676307229195 (+8 ms)Writing region close event to WAL at 1676307229198 (+3 ms)Running coprocessor post-close hooks at 1676307229203 (+5 ms)Closed at 1676307229203 2023-02-13 16:53:49,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676307222086.e0ebffe3b7f1aa1f53e67e1b20582aa7. 2023-02-13 16:53:49,303 INFO [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,45145,1676307220019; all regions closed. 2023-02-13 16:53:49,308 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33173,1676307227140; all regions closed. 2023-02-13 16:53:49,309 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37285,1676307219983; all regions closed. 2023-02-13 16:53:49,309 DEBUG [RS:2;jenkins-hbase4:45145] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs 2023-02-13 16:53:49,309 INFO [RS:2;jenkins-hbase4:45145] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C45145%2C1676307220019:(num 1676307221614) 2023-02-13 16:53:49,309 DEBUG [RS:2;jenkins-hbase4:45145] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:49,309 INFO [RS:2;jenkins-hbase4:45145] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:49,309 INFO [RS:2;jenkins-hbase4:45145] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 16:53:49,310 INFO [RS:2;jenkins-hbase4:45145] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:53:49,310 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:49,310 INFO [RS:2;jenkins-hbase4:45145] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:53:49,310 INFO [RS:2;jenkins-hbase4:45145] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:53:49,312 INFO [RS:2;jenkins-hbase4:45145] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45145 2023-02-13 16:53:49,317 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:49,317 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:49,317 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:49,317 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:49,317 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:49,317 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45145,1676307220019 2023-02-13 16:53:49,317 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7545e44f rejected from java.util.concurrent.ThreadPoolExecutor@717a7e3f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,318 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:49,318 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5f3bbdd9 rejected from java.util.concurrent.ThreadPoolExecutor@717a7e3f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,319 DEBUG [RS:3;jenkins-hbase4:33173] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs 2023-02-13 16:53:49,319 INFO [RS:3;jenkins-hbase4:33173] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33173%2C1676307227140.meta:.meta(num 1676307227734) 2023-02-13 16:53:49,320 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,45145,1676307220019] 2023-02-13 16:53:49,320 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,45145,1676307220019; numProcessing=1 2023-02-13 16:53:49,320 DEBUG [RS:1;jenkins-hbase4:37285] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs 2023-02-13 16:53:49,320 INFO [RS:1;jenkins-hbase4:37285] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37285%2C1676307219983:(num 1676307221614) 2023-02-13 16:53:49,320 DEBUG [RS:1;jenkins-hbase4:37285] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:49,320 INFO [RS:1;jenkins-hbase4:37285] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:49,321 INFO [RS:1;jenkins-hbase4:37285] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 16:53:49,321 INFO [RS:1;jenkins-hbase4:37285] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:53:49,321 INFO [RS:1;jenkins-hbase4:37285] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:53:49,321 INFO [RS:1;jenkins-hbase4:37285] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:53:49,321 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:49,323 INFO [RS:1;jenkins-hbase4:37285] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37285 2023-02-13 16:53:49,327 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/WALs/jenkins-hbase4.apache.org,33173,1676307227140/jenkins-hbase4.apache.org%2C33173%2C1676307227140.1676307227694 not finished, retry = 0 2023-02-13 16:53:49,420 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:49,420 INFO [RS:2;jenkins-hbase4:45145] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,45145,1676307220019; zookeeper connection closed. 2023-02-13 16:53:49,420 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@272f50e4 rejected from java.util.concurrent.ThreadPoolExecutor@717a7e3f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,421 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@252ac732] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@252ac732 2023-02-13 16:53:49,421 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:45145-0x1004e9a276c0003, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:49,421 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5680bb1d rejected from java.util.concurrent.ThreadPoolExecutor@717a7e3f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,422 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:49,422 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,45145,1676307220019 already deleted, retry=false 2023-02-13 16:53:49,422 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:49,422 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37285,1676307219983 2023-02-13 16:53:49,422 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,45145,1676307220019 expired; onlineServers=2 2023-02-13 16:53:49,423 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d189f42 rejected from java.util.concurrent.ThreadPoolExecutor@79270439[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,424 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37285,1676307219983] 2023-02-13 16:53:49,424 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37285,1676307219983; numProcessing=2 2023-02-13 16:53:49,425 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37285,1676307219983 already deleted, retry=false 2023-02-13 16:53:49,425 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37285,1676307219983 expired; onlineServers=1 2023-02-13 16:53:49,431 DEBUG [RS:3;jenkins-hbase4:33173] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/oldWALs 2023-02-13 16:53:49,431 INFO [RS:3;jenkins-hbase4:33173] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33173%2C1676307227140:(num 1676307227694) 2023-02-13 16:53:49,431 DEBUG [RS:3;jenkins-hbase4:33173] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:49,431 INFO [RS:3;jenkins-hbase4:33173] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:49,432 INFO [RS:3;jenkins-hbase4:33173] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:33173 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 16:53:49,432 INFO [regionserver/jenkins-hbase4:33173.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:49,433 INFO [RS:3;jenkins-hbase4:33173] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33173 2023-02-13 16:53:49,436 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33173,1676307227140 2023-02-13 16:53:49,436 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:49,436 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@709e01eb rejected from java.util.concurrent.ThreadPoolExecutor@74a19b20[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,437 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33173,1676307227140] 2023-02-13 16:53:49,437 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33173,1676307227140; numProcessing=3 2023-02-13 16:53:49,438 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33173,1676307227140 already deleted, retry=false 2023-02-13 16:53:49,439 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33173,1676307227140 expired; onlineServers=0 2023-02-13 16:53:49,439 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40475,1676307227091' ***** 2023-02-13 16:53:49,439 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 16:53:49,439 DEBUG [M:0;jenkins-hbase4:40475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d714bf5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:49,439 INFO [M:0;jenkins-hbase4:40475] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40475,1676307227091 2023-02-13 16:53:49,439 INFO [M:0;jenkins-hbase4:40475] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40475,1676307227091; all regions closed. 2023-02-13 16:53:49,439 DEBUG [M:0;jenkins-hbase4:40475] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:49,439 DEBUG [M:0;jenkins-hbase4:40475] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 16:53:49,439 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 16:53:49,439 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307227390] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307227390,5,FailOnTimeoutGroup] 2023-02-13 16:53:49,439 DEBUG [M:0;jenkins-hbase4:40475] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 16:53:49,439 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307227390] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307227390,5,FailOnTimeoutGroup] 2023-02-13 16:53:49,440 INFO [M:0;jenkins-hbase4:40475] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 16:53:49,441 INFO [M:0;jenkins-hbase4:40475] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 16:53:49,441 INFO [M:0;jenkins-hbase4:40475] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 16:53:49,441 DEBUG [M:0;jenkins-hbase4:40475] master.HMaster(1502): Stopping service threads 2023-02-13 16:53:49,441 INFO [M:0;jenkins-hbase4:40475] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 16:53:49,441 ERROR [M:0;jenkins-hbase4:40475] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-13 16:53:49,441 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:49,441 INFO [M:0;jenkins-hbase4:40475] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 16:53:49,441 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:49,442 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 16:53:49,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:49,442 DEBUG [M:0;jenkins-hbase4:40475] zookeeper.ZKUtil(398): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 16:53:49,442 WARN [M:0;jenkins-hbase4:40475] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 16:53:49,442 INFO [M:0;jenkins-hbase4:40475] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 16:53:49,442 INFO [M:0;jenkins-hbase4:40475] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 16:53:49,443 DEBUG [M:0;jenkins-hbase4:40475] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 16:53:49,443 INFO [M:0;jenkins-hbase4:40475] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:49,443 DEBUG [M:0;jenkins-hbase4:40475] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:49,443 DEBUG [M:0;jenkins-hbase4:40475] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 16:53:49,443 DEBUG [M:0;jenkins-hbase4:40475] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:49,443 INFO [M:0;jenkins-hbase4:40475] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=15.50 KB heapSize=19.41 KB 2023-02-13 16:53:49,458 INFO [M:0;jenkins-hbase4:40475] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.50 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a0816d8f773540a4a6323be185dfcde9 2023-02-13 16:53:49,464 DEBUG [M:0;jenkins-hbase4:40475] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a0816d8f773540a4a6323be185dfcde9 as hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a0816d8f773540a4a6323be185dfcde9 2023-02-13 16:53:49,470 INFO [M:0;jenkins-hbase4:40475] regionserver.HStore(1080): Added hdfs://localhost:39815/user/jenkins/test-data/626e4b04-0a8e-72da-62e0-0b42c62e2df2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a0816d8f773540a4a6323be185dfcde9, entries=5, sequenceid=186, filesize=5.3 K 2023-02-13 16:53:49,472 INFO [M:0;jenkins-hbase4:40475] regionserver.HRegion(2947): Finished flush of dataSize ~15.50 KB/15876, heapSize ~19.40 KB/19864, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 29ms, sequenceid=186, compaction requested=false 2023-02-13 16:53:49,474 INFO [M:0;jenkins-hbase4:40475] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:49,474 DEBUG [M:0;jenkins-hbase4:40475] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676307229442Disabling compacts and flushes for region at 1676307229443 (+1 ms)Disabling writes for close at 1676307229443Obtaining lock to block concurrent updates at 1676307229443Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676307229443Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=15876, getHeapSize=19864, getOffHeapSize=0, getCellsCount=49 at 1676307229444 (+1 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676307229445 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676307229445Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676307229448 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676307229448Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676307229465 (+17 ms)Finished flush of dataSize ~15.50 KB/15876, heapSize ~19.40 KB/19864, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 29ms, sequenceid=186, compaction requested=false at 1676307229472 (+7 ms)Writing region close event to WAL at 1676307229474 (+2 ms)Closed at 1676307229474 2023-02-13 16:53:49,477 INFO [M:0;jenkins-hbase4:40475] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 16:53:49,477 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:49,478 INFO [M:0;jenkins-hbase4:40475] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40475 2023-02-13 16:53:49,481 DEBUG [M:0;jenkins-hbase4:40475] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,40475,1676307227091 already deleted, retry=false 2023-02-13 16:53:49,598 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:49,598 INFO [M:0;jenkins-hbase4:40475] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40475,1676307227091; zookeeper connection closed. 2023-02-13 16:53:49,598 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3e82bbd0 rejected from java.util.concurrent.ThreadPoolExecutor@3abd38d8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 18] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,599 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): master:40475-0x1004e9a276c0009, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:49,599 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@65fd75a7 rejected from java.util.concurrent.ThreadPoolExecutor@3abd38d8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 18] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,698 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:49,698 INFO [RS:3;jenkins-hbase4:33173] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33173,1676307227140; zookeeper connection closed. 2023-02-13 16:53:49,698 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@c9b6c4 rejected from java.util.concurrent.ThreadPoolExecutor@74a19b20[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,699 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@41ac8bf3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@41ac8bf3 2023-02-13 16:53:49,699 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:33173-0x1004e9a276c000a, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:49,699 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@78d89ddf rejected from java.util.concurrent.ThreadPoolExecutor@74a19b20[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,798 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:49,798 INFO [RS:1;jenkins-hbase4:37285] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37285,1676307219983; zookeeper connection closed. 2023-02-13 16:53:49,799 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@34080093 rejected from java.util.concurrent.ThreadPoolExecutor@79270439[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,799 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@218af6b9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@218af6b9 2023-02-13 16:53:49,799 DEBUG [Listener at localhost/45541-EventThread] zookeeper.ZKWatcher(600): regionserver:37285-0x1004e9a276c0002, quorum=127.0.0.1:59338, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:49,799 ERROR [Listener at localhost/45541-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c14a090 rejected from java.util.concurrent.ThreadPoolExecutor@79270439[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:49,799 INFO [Listener at localhost/45541] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 4 regionserver(s) complete 2023-02-13 16:53:49,800 WARN [Listener at localhost/45541] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 16:53:49,804 INFO [Listener at localhost/45541] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 16:53:49,909 WARN [BP-1904083020-172.31.14.131-1676307215080 heartbeating to localhost/127.0.0.1:39815] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 16:53:49,909 WARN [BP-1904083020-172.31.14.131-1676307215080 heartbeating to localhost/127.0.0.1:39815] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1904083020-172.31.14.131-1676307215080 (Datanode Uuid 32e6136a-e7c4-43a4-9f8d-381de6a24c22) service to localhost/127.0.0.1:39815 2023-02-13 16:53:49,911 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48/dfs/data/data5/current/BP-1904083020-172.31.14.131-1676307215080] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:53:49,911 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48/dfs/data/data6/current/BP-1904083020-172.31.14.131-1676307215080] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:53:49,913 WARN [Listener at localhost/45541] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 16:53:49,915 INFO [Listener at localhost/45541] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 16:53:50,018 WARN [BP-1904083020-172.31.14.131-1676307215080 heartbeating to localhost/127.0.0.1:39815] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 16:53:50,018 WARN [BP-1904083020-172.31.14.131-1676307215080 heartbeating to localhost/127.0.0.1:39815] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1904083020-172.31.14.131-1676307215080 (Datanode Uuid e38a03ee-4b5e-4a0b-a0f9-95c3afcd0c1f) service to localhost/127.0.0.1:39815 2023-02-13 16:53:50,018 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48/dfs/data/data3/current/BP-1904083020-172.31.14.131-1676307215080] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:53:50,019 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48/dfs/data/data4/current/BP-1904083020-172.31.14.131-1676307215080] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:53:50,020 WARN [Listener at localhost/45541] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 16:53:50,022 INFO [Listener at localhost/45541] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 16:53:50,126 WARN [BP-1904083020-172.31.14.131-1676307215080 heartbeating to localhost/127.0.0.1:39815] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 16:53:50,126 WARN [BP-1904083020-172.31.14.131-1676307215080 heartbeating to localhost/127.0.0.1:39815] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1904083020-172.31.14.131-1676307215080 (Datanode Uuid f56ddadc-7b09-4b7b-8277-059e3c8c86f8) service to localhost/127.0.0.1:39815 2023-02-13 16:53:50,126 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48/dfs/data/data1/current/BP-1904083020-172.31.14.131-1676307215080] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:53:50,127 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/cluster_e6689499-d78a-152d-19b6-8d99f3413e48/dfs/data/data2/current/BP-1904083020-172.31.14.131-1676307215080] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:53:50,159 INFO [Listener at localhost/45541] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 16:53:50,277 INFO [Listener at localhost/45541] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 16:53:50,327 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-13 16:53:50,342 INFO [Listener at localhost/45541] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnSingleRSRestart Thread=102 (was 7) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:39815 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/45541 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:39815 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:39815 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:39815 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:39815 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:39815 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:39815 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:39815 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:39815 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:39815 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:39815 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:300) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:33173.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:39815 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:39815 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@2020d637 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=516 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=379 (was 357) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=1591 (was 2924) 2023-02-13 16:53:50,353 INFO [Listener at localhost/45541] hbase.ResourceChecker(147): before: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=102, OpenFileDescriptor=516, MaxFileDescriptor=60000, SystemLoadAverage=379, ProcessCount=168, AvailableMemoryMB=1591 2023-02-13 16:53:50,365 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/hadoop.log.dir so I do NOT create it in target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb 2023-02-13 16:53:50,365 WARN [Listener at localhost/45541] hbase.HBaseTestingUtility(449): hadoop.log.dir property value differs in configuration and system: Configuration=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/../logs while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/hadoop.log.dir Erasing configuration value by system value. 2023-02-13 16:53:50,366 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/hadoop.tmp.dir so I do NOT create it in target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb 2023-02-13 16:53:50,366 WARN [Listener at localhost/45541] hbase.HBaseTestingUtility(449): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-jenkins while System=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a50d8db6-ff9f-46a2-7c80-f7c4f273d08f/hadoop.tmp.dir Erasing configuration value by system value. 2023-02-13 16:53:50,366 DEBUG [Listener at localhost/45541] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb 2023-02-13 16:53:50,366 INFO [Listener at localhost/45541] master.AbstractTestRestartCluster(45): WAL splitting coordinated by zk false 2023-02-13 16:53:50,366 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 16:53:50,366 INFO [Listener at localhost/45541] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945, deleteOnExit=true 2023-02-13 16:53:50,366 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 16:53:50,366 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/test.cache.data in system properties and HBase conf 2023-02-13 16:53:50,366 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 16:53:50,366 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/hadoop.log.dir in system properties and HBase conf 2023-02-13 16:53:50,367 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 16:53:50,367 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 16:53:50,367 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 16:53:50,367 DEBUG [Listener at localhost/45541] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 16:53:50,367 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 16:53:50,367 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 16:53:50,367 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 16:53:50,368 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 16:53:50,368 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 16:53:50,368 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 16:53:50,368 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 16:53:50,368 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 16:53:50,368 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 16:53:50,368 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/nfs.dump.dir in system properties and HBase conf 2023-02-13 16:53:50,368 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/java.io.tmpdir in system properties and HBase conf 2023-02-13 16:53:50,368 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 16:53:50,369 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 16:53:50,369 INFO [Listener at localhost/45541] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 16:53:50,373 WARN [Listener at localhost/45541] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 16:53:50,373 WARN [Listener at localhost/45541] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 16:53:50,414 WARN [Listener at localhost/45541] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 16:53:50,416 INFO [Listener at localhost/45541] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 16:53:50,421 INFO [Listener at localhost/45541] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/java.io.tmpdir/Jetty_localhost_40595_hdfs____1dkbzz/webapp 2023-02-13 16:53:50,519 INFO [Listener at localhost/45541] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40595 2023-02-13 16:53:50,523 WARN [Listener at localhost/45541] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 16:53:50,523 WARN [Listener at localhost/45541] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 16:53:50,567 WARN [Listener at localhost/37465] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 16:53:50,585 WARN [Listener at localhost/37465] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 16:53:50,588 WARN [Listener at localhost/37465] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 16:53:50,589 INFO [Listener at localhost/37465] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 16:53:50,594 INFO [Listener at localhost/37465] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/java.io.tmpdir/Jetty_localhost_38913_datanode____.hdmmwz/webapp 2023-02-13 16:53:50,686 INFO [Listener at localhost/37465] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38913 2023-02-13 16:53:50,693 WARN [Listener at localhost/41673] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 16:53:50,706 WARN [Listener at localhost/41673] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 16:53:50,710 WARN [Listener at localhost/41673] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 16:53:50,711 INFO [Listener at localhost/41673] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 16:53:50,717 INFO [Listener at localhost/41673] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/java.io.tmpdir/Jetty_localhost_42215_datanode____dhsy9r/webapp 2023-02-13 16:53:50,799 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2638ddedb6ebadad: Processing first storage report for DS-9fbae8f7-cfec-455a-bf28-372062ff5846 from datanode 47024625-c8f3-4dac-b947-85fd356dcf27 2023-02-13 16:53:50,799 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2638ddedb6ebadad: from storage DS-9fbae8f7-cfec-455a-bf28-372062ff5846 node DatanodeRegistration(127.0.0.1:35077, datanodeUuid=47024625-c8f3-4dac-b947-85fd356dcf27, infoPort=44159, infoSecurePort=0, ipcPort=41673, storageInfo=lv=-57;cid=testClusterID;nsid=1687647612;c=1676307230376), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:50,799 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2638ddedb6ebadad: Processing first storage report for DS-0d6691ff-ad97-4811-922a-03bc56aca151 from datanode 47024625-c8f3-4dac-b947-85fd356dcf27 2023-02-13 16:53:50,799 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2638ddedb6ebadad: from storage DS-0d6691ff-ad97-4811-922a-03bc56aca151 node DatanodeRegistration(127.0.0.1:35077, datanodeUuid=47024625-c8f3-4dac-b947-85fd356dcf27, infoPort=44159, infoSecurePort=0, ipcPort=41673, storageInfo=lv=-57;cid=testClusterID;nsid=1687647612;c=1676307230376), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:50,815 INFO [Listener at localhost/41673] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42215 2023-02-13 16:53:50,822 WARN [Listener at localhost/42641] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 16:53:50,843 WARN [Listener at localhost/42641] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 16:53:50,846 WARN [Listener at localhost/42641] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 16:53:50,847 INFO [Listener at localhost/42641] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 16:53:50,853 INFO [Listener at localhost/42641] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/java.io.tmpdir/Jetty_localhost_37857_datanode____.fou7mr/webapp 2023-02-13 16:53:50,933 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x987c71991d6f8b32: Processing first storage report for DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e from datanode 78d69e1e-33b1-458d-a2ad-bd3e66e638b0 2023-02-13 16:53:50,933 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x987c71991d6f8b32: from storage DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e node DatanodeRegistration(127.0.0.1:43329, datanodeUuid=78d69e1e-33b1-458d-a2ad-bd3e66e638b0, infoPort=33133, infoSecurePort=0, ipcPort=42641, storageInfo=lv=-57;cid=testClusterID;nsid=1687647612;c=1676307230376), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:50,934 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x987c71991d6f8b32: Processing first storage report for DS-f8c216cf-4503-42e7-bef2-3c8162d10859 from datanode 78d69e1e-33b1-458d-a2ad-bd3e66e638b0 2023-02-13 16:53:50,934 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x987c71991d6f8b32: from storage DS-f8c216cf-4503-42e7-bef2-3c8162d10859 node DatanodeRegistration(127.0.0.1:43329, datanodeUuid=78d69e1e-33b1-458d-a2ad-bd3e66e638b0, infoPort=33133, infoSecurePort=0, ipcPort=42641, storageInfo=lv=-57;cid=testClusterID;nsid=1687647612;c=1676307230376), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:50,959 INFO [Listener at localhost/42641] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37857 2023-02-13 16:53:50,967 WARN [Listener at localhost/45077] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 16:53:51,069 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2c94a8ae8369091e: Processing first storage report for DS-dfc43236-f482-43c9-80b4-6c5055277247 from datanode e45dd6e8-bd82-41c6-a1be-6ba70c0c0a15 2023-02-13 16:53:51,069 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2c94a8ae8369091e: from storage DS-dfc43236-f482-43c9-80b4-6c5055277247 node DatanodeRegistration(127.0.0.1:39159, datanodeUuid=e45dd6e8-bd82-41c6-a1be-6ba70c0c0a15, infoPort=34317, infoSecurePort=0, ipcPort=45077, storageInfo=lv=-57;cid=testClusterID;nsid=1687647612;c=1676307230376), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:51,069 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2c94a8ae8369091e: Processing first storage report for DS-a0ea3089-f910-4e7f-ba2e-0794783a1129 from datanode e45dd6e8-bd82-41c6-a1be-6ba70c0c0a15 2023-02-13 16:53:51,069 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2c94a8ae8369091e: from storage DS-a0ea3089-f910-4e7f-ba2e-0794783a1129 node DatanodeRegistration(127.0.0.1:39159, datanodeUuid=e45dd6e8-bd82-41c6-a1be-6ba70c0c0a15, infoPort=34317, infoSecurePort=0, ipcPort=45077, storageInfo=lv=-57;cid=testClusterID;nsid=1687647612;c=1676307230376), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 16:53:51,082 DEBUG [Listener at localhost/45077] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb 2023-02-13 16:53:51,089 INFO [Listener at localhost/45077] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945/zookeeper_0, clientPort=61694, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 16:53:51,092 INFO [Listener at localhost/45077] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=61694 2023-02-13 16:53:51,092 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,093 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,114 INFO [Listener at localhost/45077] util.FSUtils(479): Created version file at hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76 with version=8 2023-02-13 16:53:51,114 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/hbase-staging 2023-02-13 16:53:51,116 INFO [Listener at localhost/45077] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:51,116 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,116 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,116 INFO [Listener at localhost/45077] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:51,116 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,116 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:51,116 INFO [Listener at localhost/45077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:51,117 INFO [Listener at localhost/45077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39061 2023-02-13 16:53:51,117 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,118 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,119 INFO [Listener at localhost/45077] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39061 connecting to ZooKeeper ensemble=127.0.0.1:61694 2023-02-13 16:53:51,132 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:390610x0, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:51,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39061-0x1004e9a5b350000 connected 2023-02-13 16:53:51,162 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:51,162 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:51,163 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:51,163 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39061 2023-02-13 16:53:51,164 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39061 2023-02-13 16:53:51,172 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39061 2023-02-13 16:53:51,172 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39061 2023-02-13 16:53:51,172 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39061 2023-02-13 16:53:51,172 INFO [Listener at localhost/45077] master.HMaster(439): hbase.rootdir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76, hbase.cluster.distributed=false 2023-02-13 16:53:51,187 INFO [Listener at localhost/45077] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:51,187 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,187 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,187 INFO [Listener at localhost/45077] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:51,187 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,187 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:51,187 INFO [Listener at localhost/45077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:51,188 INFO [Listener at localhost/45077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46119 2023-02-13 16:53:51,188 INFO [Listener at localhost/45077] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:51,189 DEBUG [Listener at localhost/45077] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:51,190 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,191 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,192 INFO [Listener at localhost/45077] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46119 connecting to ZooKeeper ensemble=127.0.0.1:61694 2023-02-13 16:53:51,196 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:461190x0, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:51,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46119-0x1004e9a5b350001 connected 2023-02-13 16:53:51,198 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:51,198 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:51,199 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:51,200 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46119 2023-02-13 16:53:51,203 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46119 2023-02-13 16:53:51,203 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46119 2023-02-13 16:53:51,203 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46119 2023-02-13 16:53:51,204 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46119 2023-02-13 16:53:51,216 INFO [Listener at localhost/45077] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:51,216 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,216 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,216 INFO [Listener at localhost/45077] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:51,216 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,216 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:51,216 INFO [Listener at localhost/45077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:51,217 INFO [Listener at localhost/45077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39253 2023-02-13 16:53:51,218 INFO [Listener at localhost/45077] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:51,219 DEBUG [Listener at localhost/45077] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:51,220 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,221 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,222 INFO [Listener at localhost/45077] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39253 connecting to ZooKeeper ensemble=127.0.0.1:61694 2023-02-13 16:53:51,228 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:392530x0, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:51,229 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:392530x0, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:51,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39253-0x1004e9a5b350002 connected 2023-02-13 16:53:51,230 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:51,231 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:51,231 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39253 2023-02-13 16:53:51,231 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39253 2023-02-13 16:53:51,232 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39253 2023-02-13 16:53:51,236 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39253 2023-02-13 16:53:51,236 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39253 2023-02-13 16:53:51,248 INFO [Listener at localhost/45077] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:51,248 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,249 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,249 INFO [Listener at localhost/45077] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:51,249 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:51,249 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:51,249 INFO [Listener at localhost/45077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:51,252 INFO [Listener at localhost/45077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39009 2023-02-13 16:53:51,252 INFO [Listener at localhost/45077] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:51,254 DEBUG [Listener at localhost/45077] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:51,254 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,255 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,256 INFO [Listener at localhost/45077] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39009 connecting to ZooKeeper ensemble=127.0.0.1:61694 2023-02-13 16:53:51,260 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:390090x0, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:51,261 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:390090x0, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:51,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39009-0x1004e9a5b350003 connected 2023-02-13 16:53:51,262 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:53:51,262 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:51,264 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39009 2023-02-13 16:53:51,264 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39009 2023-02-13 16:53:51,264 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39009 2023-02-13 16:53:51,268 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39009 2023-02-13 16:53:51,268 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39009 2023-02-13 16:53:51,269 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39061,1676307231115 2023-02-13 16:53:51,271 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 16:53:51,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39061,1676307231115 2023-02-13 16:53:51,273 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:51,273 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:51,273 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:51,273 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:51,273 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:51,275 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:51,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:51,276 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39061,1676307231115 from backup master directory 2023-02-13 16:53:51,279 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39061,1676307231115 2023-02-13 16:53:51,279 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 16:53:51,279 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:51,279 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,39061,1676307231115 2023-02-13 16:53:51,297 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/hbase.id with ID: 1d915b4a-57b4-498c-b754-ef7b45ac1780 2023-02-13 16:53:51,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:51,312 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:51,322 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x65c1ac1e to 127.0.0.1:61694 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:51,325 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28e05024, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:51,326 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:51,326 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 16:53:51,327 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:51,328 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store-tmp 2023-02-13 16:53:51,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:51,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 16:53:51,340 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:51,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:51,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 16:53:51,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:51,340 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:51,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676307231340Disabling compacts and flushes for region at 1676307231340Disabling writes for close at 1676307231340Writing region close event to WAL at 1676307231340Closed at 1676307231340 2023-02-13 16:53:51,341 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,39061,1676307231115 2023-02-13 16:53:51,344 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39061%2C1676307231115, suffix=, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,39061,1676307231115, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/oldWALs, maxLogs=10 2023-02-13 16:53:51,360 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:51,362 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:51,361 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:51,364 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,39061,1676307231115/jenkins-hbase4.apache.org%2C39061%2C1676307231115.1676307231344 2023-02-13 16:53:51,364 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK], DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK], DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK]] 2023-02-13 16:53:51,364 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:51,364 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:51,365 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:51,365 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:51,367 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:51,369 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 16:53:51,369 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 16:53:51,370 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:51,370 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:51,371 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:51,374 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:51,377 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:51,378 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64879108, jitterRate=-0.033225953578948975}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:51,378 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676307231365Initializing all the Stores at 1676307231366 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307231366Cleaning up temporary data from old regions at 1676307231371 (+5 ms)Cleaning up detritus from prior splits at 1676307231372 (+1 ms)Region opened successfully at 1676307231378 (+6 ms) 2023-02-13 16:53:51,378 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 16:53:51,380 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 16:53:51,380 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 16:53:51,380 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 16:53:51,380 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 16:53:51,380 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-02-13 16:53:51,381 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 16:53:51,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 16:53:51,382 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 16:53:51,393 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 16:53:51,393 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 16:53:51,394 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 16:53:51,394 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 16:53:51,394 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 16:53:51,396 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:51,396 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 16:53:51,397 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 16:53:51,397 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 16:53:51,400 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:51,400 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:51,400 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:51,400 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:51,400 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 16:53:51,401 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,39061,1676307231115, sessionid=0x1004e9a5b350000, setting cluster-up flag (Was=false) 2023-02-13 16:53:51,405 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:51,410 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 16:53:51,411 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39061,1676307231115 2023-02-13 16:53:51,414 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:51,419 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 16:53:51,420 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39061,1676307231115 2023-02-13 16:53:51,420 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.hbase-snapshot/.tmp 2023-02-13 16:53:51,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 16:53:51,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:51,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:51,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:51,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:51,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 16:53:51,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:51,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,427 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676307261427 2023-02-13 16:53:51,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 16:53:51,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 16:53:51,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 16:53:51,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 16:53:51,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 16:53:51,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 16:53:51,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 16:53:51,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 16:53:51,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 16:53:51,429 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 16:53:51,429 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 16:53:51,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 16:53:51,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 16:53:51,429 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307231429,5,FailOnTimeoutGroup] 2023-02-13 16:53:51,429 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307231429,5,FailOnTimeoutGroup] 2023-02-13 16:53:51,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 16:53:51,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,430 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:51,443 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:51,443 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:51,443 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76 2023-02-13 16:53:51,453 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:51,454 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 16:53:51,456 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info 2023-02-13 16:53:51,456 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 16:53:51,457 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:51,457 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 16:53:51,458 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:51,459 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 16:53:51,459 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:51,459 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 16:53:51,461 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/table 2023-02-13 16:53:51,461 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 16:53:51,462 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:51,462 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740 2023-02-13 16:53:51,463 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740 2023-02-13 16:53:51,465 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 16:53:51,467 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 16:53:51,470 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(952): ClusterId : 1d915b4a-57b4-498c-b754-ef7b45ac1780 2023-02-13 16:53:51,470 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(952): ClusterId : 1d915b4a-57b4-498c-b754-ef7b45ac1780 2023-02-13 16:53:51,470 DEBUG [RS:1;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:51,470 INFO [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(952): ClusterId : 1d915b4a-57b4-498c-b754-ef7b45ac1780 2023-02-13 16:53:51,470 DEBUG [RS:0;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:51,470 DEBUG [RS:2;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:51,475 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:51,476 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64835135, jitterRate=-0.03388120234012604}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 16:53:51,476 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676307231453Initializing all the Stores at 1676307231454 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307231454Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307231454Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307231454Cleaning up temporary data from old regions at 1676307231464 (+10 ms)Cleaning up detritus from prior splits at 1676307231464Region opened successfully at 1676307231476 (+12 ms) 2023-02-13 16:53:51,476 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 16:53:51,476 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 16:53:51,476 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 16:53:51,477 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 16:53:51,477 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 16:53:51,477 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 16:53:51,477 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676307231476Disabling compacts and flushes for region at 1676307231476Disabling writes for close at 1676307231477 (+1 ms)Writing region close event to WAL at 1676307231477Closed at 1676307231477 2023-02-13 16:53:51,478 DEBUG [RS:1;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:51,478 DEBUG [RS:1;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:51,479 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 16:53:51,479 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 16:53:51,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 16:53:51,481 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 16:53:51,481 DEBUG [RS:2;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:51,482 DEBUG [RS:2;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:51,482 DEBUG [RS:0;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:51,482 DEBUG [RS:0;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:51,484 DEBUG [RS:1;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:51,484 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 16:53:51,485 DEBUG [RS:1;jenkins-hbase4:39253] zookeeper.ReadOnlyZKClient(139): Connect 0x513051d3 to 127.0.0.1:61694 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:51,487 DEBUG [RS:2;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:51,487 DEBUG [RS:0;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:51,489 DEBUG [RS:2;jenkins-hbase4:39009] zookeeper.ReadOnlyZKClient(139): Connect 0x595aaab6 to 127.0.0.1:61694 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:51,489 DEBUG [RS:0;jenkins-hbase4:46119] zookeeper.ReadOnlyZKClient(139): Connect 0x2922c961 to 127.0.0.1:61694 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:51,501 DEBUG [RS:1;jenkins-hbase4:39253] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ff7b585, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:51,502 DEBUG [RS:1;jenkins-hbase4:39253] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ecabfbe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:51,503 DEBUG [RS:2;jenkins-hbase4:39009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@579d4fed, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:51,503 DEBUG [RS:0;jenkins-hbase4:46119] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27e53235, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:51,504 DEBUG [RS:2;jenkins-hbase4:39009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53a2c1c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:51,504 DEBUG [RS:0;jenkins-hbase4:46119] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23daa211, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:51,514 DEBUG [RS:1;jenkins-hbase4:39253] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:39253 2023-02-13 16:53:51,514 INFO [RS:1;jenkins-hbase4:39253] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:51,514 INFO [RS:1;jenkins-hbase4:39253] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:51,514 DEBUG [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:51,514 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39061,1676307231115 with isa=jenkins-hbase4.apache.org/172.31.14.131:39253, startcode=1676307231215 2023-02-13 16:53:51,515 DEBUG [RS:1;jenkins-hbase4:39253] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:51,515 DEBUG [RS:0;jenkins-hbase4:46119] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:46119 2023-02-13 16:53:51,515 INFO [RS:0;jenkins-hbase4:46119] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:51,515 INFO [RS:0;jenkins-hbase4:46119] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:51,515 DEBUG [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:51,516 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39061,1676307231115 with isa=jenkins-hbase4.apache.org/172.31.14.131:46119, startcode=1676307231186 2023-02-13 16:53:51,516 DEBUG [RS:2;jenkins-hbase4:39009] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:39009 2023-02-13 16:53:51,516 DEBUG [RS:0;jenkins-hbase4:46119] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:51,516 INFO [RS:2;jenkins-hbase4:39009] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:51,516 INFO [RS:2;jenkins-hbase4:39009] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:51,516 DEBUG [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:51,517 INFO [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39061,1676307231115 with isa=jenkins-hbase4.apache.org/172.31.14.131:39009, startcode=1676307231248 2023-02-13 16:53:51,517 DEBUG [RS:2;jenkins-hbase4:39009] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:51,517 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54875, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.6 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:51,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39061] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:51,520 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42749, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:51,520 INFO [RS-EventLoopGroup-9-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57117, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.7 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:51,520 DEBUG [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76 2023-02-13 16:53:51,521 DEBUG [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37465 2023-02-13 16:53:51,521 DEBUG [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:51,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39061] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:51,521 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:51,522 DEBUG [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76 2023-02-13 16:53:51,522 DEBUG [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37465 2023-02-13 16:53:51,522 DEBUG [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76 2023-02-13 16:53:51,522 DEBUG [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:51,522 DEBUG [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37465 2023-02-13 16:53:51,522 DEBUG [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:51,523 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:51,527 DEBUG [RS:1;jenkins-hbase4:39253] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:51,527 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39253,1676307231215] 2023-02-13 16:53:51,528 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39009,1676307231248] 2023-02-13 16:53:51,528 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46119,1676307231186] 2023-02-13 16:53:51,528 WARN [RS:1;jenkins-hbase4:39253] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:51,528 DEBUG [RS:2;jenkins-hbase4:39009] zookeeper.ZKUtil(162): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:51,528 INFO [RS:1;jenkins-hbase4:39253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:51,528 WARN [RS:2;jenkins-hbase4:39009] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:51,528 DEBUG [RS:0;jenkins-hbase4:46119] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:51,528 DEBUG [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:51,528 INFO [RS:2;jenkins-hbase4:39009] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:51,528 WARN [RS:0;jenkins-hbase4:46119] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:51,528 DEBUG [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:51,528 INFO [RS:0;jenkins-hbase4:46119] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:51,528 DEBUG [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:51,535 DEBUG [RS:2;jenkins-hbase4:39009] zookeeper.ZKUtil(162): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:51,535 DEBUG [RS:1;jenkins-hbase4:39253] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:51,535 DEBUG [RS:0;jenkins-hbase4:46119] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:51,536 DEBUG [RS:2;jenkins-hbase4:39009] zookeeper.ZKUtil(162): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:51,536 DEBUG [RS:1;jenkins-hbase4:39253] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:51,536 DEBUG [RS:0;jenkins-hbase4:46119] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:51,536 DEBUG [RS:2;jenkins-hbase4:39009] zookeeper.ZKUtil(162): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:51,536 DEBUG [RS:1;jenkins-hbase4:39253] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:51,537 DEBUG [RS:0;jenkins-hbase4:46119] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:51,537 DEBUG [RS:2;jenkins-hbase4:39009] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:51,537 INFO [RS:2;jenkins-hbase4:39009] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:51,537 DEBUG [RS:1;jenkins-hbase4:39253] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:51,537 DEBUG [RS:0;jenkins-hbase4:46119] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:51,539 INFO [RS:0;jenkins-hbase4:46119] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:51,539 INFO [RS:1;jenkins-hbase4:39253] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:51,539 INFO [RS:2;jenkins-hbase4:39009] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:51,539 INFO [RS:2;jenkins-hbase4:39009] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:51,539 INFO [RS:2;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,540 INFO [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:51,541 INFO [RS:0;jenkins-hbase4:46119] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:51,541 INFO [RS:0;jenkins-hbase4:46119] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:51,541 INFO [RS:0;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,541 INFO [RS:2;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,542 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:51,542 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,542 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,542 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,542 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,542 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,542 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:51,543 INFO [RS:1;jenkins-hbase4:39253] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:51,543 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,543 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,543 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,543 DEBUG [RS:2;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,546 INFO [RS:1;jenkins-hbase4:39253] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:51,546 INFO [RS:1;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,546 INFO [RS:0;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,547 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:51,548 INFO [RS:2;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,549 INFO [RS:2;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,549 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,549 INFO [RS:2;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,549 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,549 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,549 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,549 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,550 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:51,550 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,550 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,550 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,550 DEBUG [RS:0;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,552 INFO [RS:1;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,554 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,554 INFO [RS:0;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,554 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,554 INFO [RS:0;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,554 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,554 INFO [RS:0;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,554 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,554 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,555 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:51,555 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,555 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,555 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,555 DEBUG [RS:1;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:51,560 INFO [RS:1;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,560 INFO [RS:1;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,560 INFO [RS:1;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,566 INFO [RS:2;jenkins-hbase4:39009] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:51,566 INFO [RS:2;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39009,1676307231248-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,570 INFO [RS:0;jenkins-hbase4:46119] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:51,570 INFO [RS:0;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46119,1676307231186-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,575 INFO [RS:1;jenkins-hbase4:39253] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:51,575 INFO [RS:1;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39253,1676307231215-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,581 INFO [RS:0;jenkins-hbase4:46119] regionserver.Replication(203): jenkins-hbase4.apache.org,46119,1676307231186 started 2023-02-13 16:53:51,581 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46119,1676307231186, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46119, sessionid=0x1004e9a5b350001 2023-02-13 16:53:51,581 DEBUG [RS:0;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:51,581 DEBUG [RS:0;jenkins-hbase4:46119] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:51,582 DEBUG [RS:0;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46119,1676307231186' 2023-02-13 16:53:51,582 DEBUG [RS:0;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:51,582 DEBUG [RS:0;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:51,582 DEBUG [RS:0;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:51,582 INFO [RS:2;jenkins-hbase4:39009] regionserver.Replication(203): jenkins-hbase4.apache.org,39009,1676307231248 started 2023-02-13 16:53:51,582 DEBUG [RS:0;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:51,582 INFO [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39009,1676307231248, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39009, sessionid=0x1004e9a5b350003 2023-02-13 16:53:51,582 DEBUG [RS:0;jenkins-hbase4:46119] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:51,583 DEBUG [RS:0;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46119,1676307231186' 2023-02-13 16:53:51,583 DEBUG [RS:0;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:51,583 DEBUG [RS:2;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:51,583 DEBUG [RS:2;jenkins-hbase4:39009] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:51,583 DEBUG [RS:2;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39009,1676307231248' 2023-02-13 16:53:51,583 DEBUG [RS:2;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:51,583 DEBUG [RS:0;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:51,583 DEBUG [RS:0;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:51,583 DEBUG [RS:2;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:51,583 INFO [RS:0;jenkins-hbase4:46119] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:51,583 INFO [RS:0;jenkins-hbase4:46119] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:51,584 DEBUG [RS:2;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:51,584 DEBUG [RS:2;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:51,584 DEBUG [RS:2;jenkins-hbase4:39009] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:51,584 DEBUG [RS:2;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39009,1676307231248' 2023-02-13 16:53:51,584 DEBUG [RS:2;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:51,584 DEBUG [RS:2;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:51,584 DEBUG [RS:2;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:51,584 INFO [RS:2;jenkins-hbase4:39009] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:51,584 INFO [RS:2;jenkins-hbase4:39009] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:51,589 INFO [RS:1;jenkins-hbase4:39253] regionserver.Replication(203): jenkins-hbase4.apache.org,39253,1676307231215 started 2023-02-13 16:53:51,589 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39253,1676307231215, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39253, sessionid=0x1004e9a5b350002 2023-02-13 16:53:51,589 DEBUG [RS:1;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:51,589 DEBUG [RS:1;jenkins-hbase4:39253] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:51,589 DEBUG [RS:1;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39253,1676307231215' 2023-02-13 16:53:51,589 DEBUG [RS:1;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:51,589 DEBUG [RS:1;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:51,590 DEBUG [RS:1;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:51,590 DEBUG [RS:1;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:51,590 DEBUG [RS:1;jenkins-hbase4:39253] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:51,590 DEBUG [RS:1;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39253,1676307231215' 2023-02-13 16:53:51,590 DEBUG [RS:1;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:51,590 DEBUG [RS:1;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:51,591 DEBUG [RS:1;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:51,591 INFO [RS:1;jenkins-hbase4:39253] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:51,591 INFO [RS:1;jenkins-hbase4:39253] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:51,635 DEBUG [jenkins-hbase4:39061] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 16:53:51,635 DEBUG [jenkins-hbase4:39061] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:51,637 DEBUG [jenkins-hbase4:39061] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:51,637 DEBUG [jenkins-hbase4:39061] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:51,637 DEBUG [jenkins-hbase4:39061] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:51,637 DEBUG [jenkins-hbase4:39061] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:51,638 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46119,1676307231186, state=OPENING 2023-02-13 16:53:51,640 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 16:53:51,641 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:51,641 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 16:53:51,641 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46119,1676307231186}] 2023-02-13 16:53:51,688 INFO [RS:0;jenkins-hbase4:46119] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46119%2C1676307231186, suffix=, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs, maxLogs=32 2023-02-13 16:53:51,689 INFO [RS:2;jenkins-hbase4:39009] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39009%2C1676307231248, suffix=, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307231248, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs, maxLogs=32 2023-02-13 16:53:51,693 INFO [RS:1;jenkins-hbase4:39253] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39253%2C1676307231215, suffix=, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307231215, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs, maxLogs=32 2023-02-13 16:53:51,713 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:51,713 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:51,713 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:51,715 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:51,716 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:51,716 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:51,720 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:51,720 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:51,720 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:51,721 INFO [RS:2;jenkins-hbase4:39009] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307231248/jenkins-hbase4.apache.org%2C39009%2C1676307231248.1676307231691 2023-02-13 16:53:51,721 DEBUG [RS:2;jenkins-hbase4:39009] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK], DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK], DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK]] 2023-02-13 16:53:51,722 INFO [RS:1;jenkins-hbase4:39253] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307231215/jenkins-hbase4.apache.org%2C39253%2C1676307231215.1676307231694 2023-02-13 16:53:51,722 DEBUG [RS:1;jenkins-hbase4:39253] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK], DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK], DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK]] 2023-02-13 16:53:51,723 INFO [RS:0;jenkins-hbase4:46119] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186/jenkins-hbase4.apache.org%2C46119%2C1676307231186.1676307231691 2023-02-13 16:53:51,723 DEBUG [RS:0;jenkins-hbase4:46119] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK], DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK], DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK]] 2023-02-13 16:53:51,796 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:51,796 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:51,798 INFO [RS-EventLoopGroup-10-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54406, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 16:53:51,802 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 16:53:51,802 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:51,803 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46119%2C1676307231186.meta, suffix=.meta, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs, maxLogs=32 2023-02-13 16:53:51,819 DEBUG [RS-EventLoopGroup-12-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:51,819 DEBUG [RS-EventLoopGroup-12-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:51,819 DEBUG [RS-EventLoopGroup-12-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:51,822 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186/jenkins-hbase4.apache.org%2C46119%2C1676307231186.meta.1676307231804.meta 2023-02-13 16:53:51,822 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK], DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK], DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK]] 2023-02-13 16:53:51,822 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:51,822 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 16:53:51,823 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 16:53:51,823 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 16:53:51,823 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 16:53:51,823 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:51,823 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 16:53:51,823 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 16:53:51,825 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 16:53:51,826 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info 2023-02-13 16:53:51,826 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info 2023-02-13 16:53:51,827 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 16:53:51,827 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:51,828 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 16:53:51,828 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:51,829 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:51,829 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 16:53:51,829 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:51,830 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 16:53:51,830 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/table 2023-02-13 16:53:51,830 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/table 2023-02-13 16:53:51,831 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 16:53:51,831 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:51,832 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740 2023-02-13 16:53:51,833 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740 2023-02-13 16:53:51,836 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 16:53:51,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 16:53:51,838 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70822878, jitterRate=0.05534312129020691}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 16:53:51,839 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676307231823Writing region info on filesystem at 1676307231823Initializing all the Stores at 1676307231824 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307231824Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307231825 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307231825Cleaning up temporary data from old regions at 1676307231835 (+10 ms)Cleaning up detritus from prior splits at 1676307231835Running coprocessor post-open hooks at 1676307231838 (+3 ms)Region opened successfully at 1676307231838 2023-02-13 16:53:51,840 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676307231796 2023-02-13 16:53:51,845 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 16:53:51,846 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 16:53:51,846 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46119,1676307231186, state=OPEN 2023-02-13 16:53:51,849 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 16:53:51,850 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 16:53:51,853 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 16:53:51,853 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46119,1676307231186 in 209 msec 2023-02-13 16:53:51,855 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 16:53:51,855 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 374 msec 2023-02-13 16:53:51,858 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 435 msec 2023-02-13 16:53:51,858 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676307231271, completionTime=-1 2023-02-13 16:53:51,858 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 16:53:51,858 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 16:53:51,862 DEBUG [hconnection-0x52c1e039-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:51,864 INFO [RS-EventLoopGroup-10-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54418, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:53:51,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 16:53:51,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676307291865 2023-02-13 16:53:51,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676307351865 2023-02-13 16:53:51,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 7 msec 2023-02-13 16:53:51,874 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39061,1676307231115-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,874 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39061,1676307231115-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,874 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39061,1676307231115-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,874 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39061, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,874 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:51,874 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 16:53:51,874 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:51,875 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 16:53:51,875 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 16:53:51,877 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 16:53:51,878 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 16:53:51,879 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:51,880 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb empty. 2023-02-13 16:53:51,880 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:51,880 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 16:53:51,894 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:51,895 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 745e4bc8b1326452bfdd5dbda9c99aeb, NAME => 'hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp 2023-02-13 16:53:51,906 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:51,907 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 745e4bc8b1326452bfdd5dbda9c99aeb, disabling compactions & flushes 2023-02-13 16:53:51,907 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:51,907 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:51,907 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. after waiting 0 ms 2023-02-13 16:53:51,907 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:51,907 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:51,907 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 745e4bc8b1326452bfdd5dbda9c99aeb: Waiting for close lock at 1676307231907Disabling compacts and flushes for region at 1676307231907Disabling writes for close at 1676307231907Writing region close event to WAL at 1676307231907Closed at 1676307231907 2023-02-13 16:53:51,910 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 16:53:51,911 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307231910"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307231910"}]},"ts":"1676307231910"} 2023-02-13 16:53:51,913 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 16:53:51,914 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 16:53:51,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307231914"}]},"ts":"1676307231914"} 2023-02-13 16:53:51,916 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 16:53:51,920 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:51,921 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:51,921 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:51,921 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:51,921 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:51,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=745e4bc8b1326452bfdd5dbda9c99aeb, ASSIGN}] 2023-02-13 16:53:51,924 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=745e4bc8b1326452bfdd5dbda9c99aeb, ASSIGN 2023-02-13 16:53:51,926 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=745e4bc8b1326452bfdd5dbda9c99aeb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46119,1676307231186; forceNewPlan=false, retain=false 2023-02-13 16:53:52,076 INFO [jenkins-hbase4:39061] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:52,077 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=745e4bc8b1326452bfdd5dbda9c99aeb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:52,077 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307232077"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307232077"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307232077"}]},"ts":"1676307232077"} 2023-02-13 16:53:52,080 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 745e4bc8b1326452bfdd5dbda9c99aeb, server=jenkins-hbase4.apache.org,46119,1676307231186}] 2023-02-13 16:53:52,236 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:52,237 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 745e4bc8b1326452bfdd5dbda9c99aeb, NAME => 'hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:52,237 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:52,237 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:52,237 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:52,237 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:52,238 INFO [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:52,240 DEBUG [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/info 2023-02-13 16:53:52,240 DEBUG [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/info 2023-02-13 16:53:52,240 INFO [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 745e4bc8b1326452bfdd5dbda9c99aeb columnFamilyName info 2023-02-13 16:53:52,241 INFO [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] regionserver.HStore(310): Store=745e4bc8b1326452bfdd5dbda9c99aeb/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:52,242 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:52,242 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:52,244 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:52,246 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:52,247 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 745e4bc8b1326452bfdd5dbda9c99aeb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62212339, jitterRate=-0.0729639083147049}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:52,247 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 745e4bc8b1326452bfdd5dbda9c99aeb: Running coprocessor pre-open hook at 1676307232237Writing region info on filesystem at 1676307232237Initializing all the Stores at 1676307232238 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307232238Cleaning up temporary data from old regions at 1676307232242 (+4 ms)Cleaning up detritus from prior splits at 1676307232243 (+1 ms)Running coprocessor post-open hooks at 1676307232247 (+4 ms)Region opened successfully at 1676307232247 2023-02-13 16:53:52,248 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb., pid=6, masterSystemTime=1676307232232 2023-02-13 16:53:52,250 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:52,250 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:52,251 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=745e4bc8b1326452bfdd5dbda9c99aeb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:52,251 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307232251"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307232251"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307232251"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307232251"}]},"ts":"1676307232251"} 2023-02-13 16:53:52,256 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 16:53:52,257 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 745e4bc8b1326452bfdd5dbda9c99aeb, server=jenkins-hbase4.apache.org,46119,1676307231186 in 174 msec 2023-02-13 16:53:52,260 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 16:53:52,260 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=745e4bc8b1326452bfdd5dbda9c99aeb, ASSIGN in 336 msec 2023-02-13 16:53:52,261 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 16:53:52,261 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307232261"}]},"ts":"1676307232261"} 2023-02-13 16:53:52,263 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 16:53:52,267 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 16:53:52,268 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 393 msec 2023-02-13 16:53:52,276 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 16:53:52,278 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 16:53:52,278 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:52,282 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 16:53:52,293 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 16:53:52,297 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 14 msec 2023-02-13 16:53:52,304 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 16:53:52,313 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 16:53:52,317 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 12 msec 2023-02-13 16:53:52,335 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 16:53:52,337 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 16:53:52,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 1.058sec 2023-02-13 16:53:52,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 16:53:52,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 16:53:52,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 16:53:52,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39061,1676307231115-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 16:53:52,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39061,1676307231115-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 16:53:52,339 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 16:53:52,376 DEBUG [Listener at localhost/45077] zookeeper.ReadOnlyZKClient(139): Connect 0x58661f63 to 127.0.0.1:61694 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:52,381 DEBUG [Listener at localhost/45077] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43f6aee4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:52,383 DEBUG [hconnection-0x77a00002-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:52,385 INFO [RS-EventLoopGroup-10-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54432, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:53:52,386 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39061,1676307231115 2023-02-13 16:53:52,389 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-13 16:53:52,389 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:52,390 INFO [Listener at localhost/45077] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-02-13 16:53:52,390 INFO [Listener at localhost/45077] master.TestRetainAssignmentOnRestart(212): Creating tables 2023-02-13 16:53:52,392 DEBUG [Listener at localhost/45077] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 16:53:52,394 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54920, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 16:53:52,396 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:52,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableOne 2023-02-13 16:53:52,399 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 16:53:52,399 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableOne" procId is: 9 2023-02-13 16:53:52,400 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 16:53:52,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 16:53:52,402 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:52,402 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca empty. 2023-02-13 16:53:52,403 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:52,403 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived restartTableOne regions 2023-02-13 16:53:52,417 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableOne/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:52,418 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(7671): creating {ENCODED => c49f69a5b49e9a6b371b23ed03123cca, NAME => 'restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableOne', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp 2023-02-13 16:53:52,434 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:52,434 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1603): Closing c49f69a5b49e9a6b371b23ed03123cca, disabling compactions & flushes 2023-02-13 16:53:52,434 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:52,434 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:52,434 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. after waiting 0 ms 2023-02-13 16:53:52,434 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:52,435 INFO [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1837): Closed restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:52,435 DEBUG [RegionOpenAndInit-restartTableOne-pool-0] regionserver.HRegion(1557): Region close journal for c49f69a5b49e9a6b371b23ed03123cca: Waiting for close lock at 1676307232434Disabling compacts and flushes for region at 1676307232434Disabling writes for close at 1676307232434Writing region close event to WAL at 1676307232434Closed at 1676307232434 2023-02-13 16:53:52,437 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 16:53:52,438 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307232438"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307232438"}]},"ts":"1676307232438"} 2023-02-13 16:53:52,440 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 16:53:52,441 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 16:53:52,441 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307232441"}]},"ts":"1676307232441"} 2023-02-13 16:53:52,442 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLING in hbase:meta 2023-02-13 16:53:52,447 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:52,447 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:52,447 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:52,447 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:52,447 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:52,448 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=c49f69a5b49e9a6b371b23ed03123cca, ASSIGN}] 2023-02-13 16:53:52,449 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=c49f69a5b49e9a6b371b23ed03123cca, ASSIGN 2023-02-13 16:53:52,450 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=c49f69a5b49e9a6b371b23ed03123cca, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39009,1676307231248; forceNewPlan=false, retain=false 2023-02-13 16:53:52,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 16:53:52,601 INFO [jenkins-hbase4:39061] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:52,602 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c49f69a5b49e9a6b371b23ed03123cca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:52,602 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307232602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307232602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307232602"}]},"ts":"1676307232602"} 2023-02-13 16:53:52,604 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure c49f69a5b49e9a6b371b23ed03123cca, server=jenkins-hbase4.apache.org,39009,1676307231248}] 2023-02-13 16:53:52,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 16:53:52,757 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:52,758 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:52,763 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43458, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 16:53:52,768 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:52,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c49f69a5b49e9a6b371b23ed03123cca, NAME => 'restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:52,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:52,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:52,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:52,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:52,770 INFO [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:52,771 DEBUG [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca/family 2023-02-13 16:53:52,771 DEBUG [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca/family 2023-02-13 16:53:52,772 INFO [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c49f69a5b49e9a6b371b23ed03123cca columnFamilyName family 2023-02-13 16:53:52,772 INFO [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] regionserver.HStore(310): Store=c49f69a5b49e9a6b371b23ed03123cca/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:52,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:52,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:52,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:52,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:52,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c49f69a5b49e9a6b371b23ed03123cca; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64541195, jitterRate=-0.03826124966144562}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:52,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c49f69a5b49e9a6b371b23ed03123cca: Running coprocessor pre-open hook at 1676307232768Writing region info on filesystem at 1676307232768Initializing all the Stores at 1676307232769 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307232769Cleaning up temporary data from old regions at 1676307232774 (+5 ms)Cleaning up detritus from prior splits at 1676307232774Running coprocessor post-open hooks at 1676307232779 (+5 ms)Region opened successfully at 1676307232779 2023-02-13 16:53:52,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca., pid=11, masterSystemTime=1676307232757 2023-02-13 16:53:52,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:52,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:52,784 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c49f69a5b49e9a6b371b23ed03123cca, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:52,785 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307232784"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307232784"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307232784"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307232784"}]},"ts":"1676307232784"} 2023-02-13 16:53:52,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-13 16:53:52,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure c49f69a5b49e9a6b371b23ed03123cca, server=jenkins-hbase4.apache.org,39009,1676307231248 in 183 msec 2023-02-13 16:53:52,792 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-13 16:53:52,792 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=c49f69a5b49e9a6b371b23ed03123cca, ASSIGN in 341 msec 2023-02-13 16:53:52,792 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 16:53:52,793 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableOne","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307232793"}]},"ts":"1676307232793"} 2023-02-13 16:53:52,794 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=restartTableOne, state=ENABLED in hbase:meta 2023-02-13 16:53:52,797 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableOne execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 16:53:52,798 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne in 401 msec 2023-02-13 16:53:53,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 16:53:53,005 INFO [Listener at localhost/45077] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableOne, procId: 9 completed 2023-02-13 16:53:53,006 DEBUG [Listener at localhost/45077] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableOne get assigned. Timeout = 60000ms 2023-02-13 16:53:53,006 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:53,008 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(3484): All regions for table restartTableOne assigned to meta. Checking AM states. 2023-02-13 16:53:53,009 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:53,009 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(3504): All regions for table restartTableOne assigned. 2023-02-13 16:53:53,010 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:53,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableTwo 2023-02-13 16:53:53,013 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 16:53:53,014 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableTwo" procId is: 12 2023-02-13 16:53:53,014 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 16:53:53,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 16:53:53,016 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:53,017 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285 empty. 2023-02-13 16:53:53,017 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:53,017 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived restartTableTwo regions 2023-02-13 16:53:53,033 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableTwo/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:53,035 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(7671): creating {ENCODED => b1fad04018b6f3dcf4291cca06397285, NAME => 'restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableTwo', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp 2023-02-13 16:53:53,047 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:53,047 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1603): Closing b1fad04018b6f3dcf4291cca06397285, disabling compactions & flushes 2023-02-13 16:53:53,047 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:53,047 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:53,048 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. after waiting 0 ms 2023-02-13 16:53:53,048 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:53,048 INFO [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:53,048 DEBUG [RegionOpenAndInit-restartTableTwo-pool-0] regionserver.HRegion(1557): Region close journal for b1fad04018b6f3dcf4291cca06397285: Waiting for close lock at 1676307233047Disabling compacts and flushes for region at 1676307233047Disabling writes for close at 1676307233048 (+1 ms)Writing region close event to WAL at 1676307233048Closed at 1676307233048 2023-02-13 16:53:53,050 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 16:53:53,051 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307233051"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307233051"}]},"ts":"1676307233051"} 2023-02-13 16:53:53,053 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 16:53:53,053 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 16:53:53,054 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307233053"}]},"ts":"1676307233053"} 2023-02-13 16:53:53,055 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLING in hbase:meta 2023-02-13 16:53:53,059 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:53,060 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:53,060 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:53,060 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:53,060 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:53,060 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=b1fad04018b6f3dcf4291cca06397285, ASSIGN}] 2023-02-13 16:53:53,062 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=b1fad04018b6f3dcf4291cca06397285, ASSIGN 2023-02-13 16:53:53,063 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=b1fad04018b6f3dcf4291cca06397285, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39253,1676307231215; forceNewPlan=false, retain=false 2023-02-13 16:53:53,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 16:53:53,213 INFO [jenkins-hbase4:39061] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:53,214 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b1fad04018b6f3dcf4291cca06397285, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:53,214 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307233214"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307233214"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307233214"}]},"ts":"1676307233214"} 2023-02-13 16:53:53,217 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure b1fad04018b6f3dcf4291cca06397285, server=jenkins-hbase4.apache.org,39253,1676307231215}] 2023-02-13 16:53:53,265 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-13 16:53:53,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 16:53:53,370 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:53,370 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:53,372 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44268, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 16:53:53,376 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:53,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b1fad04018b6f3dcf4291cca06397285, NAME => 'restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:53,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:53,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:53,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:53,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:53,378 INFO [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:53,379 DEBUG [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285/family 2023-02-13 16:53:53,379 DEBUG [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285/family 2023-02-13 16:53:53,380 INFO [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b1fad04018b6f3dcf4291cca06397285 columnFamilyName family 2023-02-13 16:53:53,380 INFO [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] regionserver.HStore(310): Store=b1fad04018b6f3dcf4291cca06397285/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:53,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:53,382 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:53,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:53,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:53,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b1fad04018b6f3dcf4291cca06397285; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70160014, jitterRate=0.045465677976608276}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:53,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b1fad04018b6f3dcf4291cca06397285: Running coprocessor pre-open hook at 1676307233377Writing region info on filesystem at 1676307233377Initializing all the Stores at 1676307233377Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307233378 (+1 ms)Cleaning up temporary data from old regions at 1676307233382 (+4 ms)Cleaning up detritus from prior splits at 1676307233383 (+1 ms)Running coprocessor post-open hooks at 1676307233388 (+5 ms)Region opened successfully at 1676307233388 2023-02-13 16:53:53,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285., pid=14, masterSystemTime=1676307233370 2023-02-13 16:53:53,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:53,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:53,394 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b1fad04018b6f3dcf4291cca06397285, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:53,394 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307233394"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307233394"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307233394"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307233394"}]},"ts":"1676307233394"} 2023-02-13 16:53:53,398 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-13 16:53:53,398 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure b1fad04018b6f3dcf4291cca06397285, server=jenkins-hbase4.apache.org,39253,1676307231215 in 179 msec 2023-02-13 16:53:53,400 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-13 16:53:53,400 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=b1fad04018b6f3dcf4291cca06397285, ASSIGN in 338 msec 2023-02-13 16:53:53,401 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 16:53:53,401 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableTwo","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307233401"}]},"ts":"1676307233401"} 2023-02-13 16:53:53,402 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=restartTableTwo, state=ENABLED in hbase:meta 2023-02-13 16:53:53,405 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableTwo execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 16:53:53,406 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo in 394 msec 2023-02-13 16:53:53,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-02-13 16:53:53,620 INFO [Listener at localhost/45077] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableTwo, procId: 12 completed 2023-02-13 16:53:53,620 DEBUG [Listener at localhost/45077] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableTwo get assigned. Timeout = 60000ms 2023-02-13 16:53:53,620 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:53,623 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(3484): All regions for table restartTableTwo assigned to meta. Checking AM states. 2023-02-13 16:53:53,623 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:53,623 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(3504): All regions for table restartTableTwo assigned. 2023-02-13 16:53:53,625 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:53,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=restartTableThree 2023-02-13 16:53:53,628 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 16:53:53,628 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "restartTableThree" procId is: 15 2023-02-13 16:53:53,629 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 16:53:53,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 16:53:53,631 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:53,631 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b empty. 2023-02-13 16:53:53,632 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:53,632 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived restartTableThree regions 2023-02-13 16:53:53,646 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp/data/default/restartTableThree/.tabledesc/.tableinfo.0000000001 2023-02-13 16:53:53,648 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(7671): creating {ENCODED => 82053ffc9c655280d2c11277837c834b, NAME => 'restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='restartTableThree', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.tmp 2023-02-13 16:53:53,661 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:53,661 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1603): Closing 82053ffc9c655280d2c11277837c834b, disabling compactions & flushes 2023-02-13 16:53:53,661 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:53,661 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:53,661 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. after waiting 0 ms 2023-02-13 16:53:53,661 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:53,661 INFO [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1837): Closed restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:53,661 DEBUG [RegionOpenAndInit-restartTableThree-pool-0] regionserver.HRegion(1557): Region close journal for 82053ffc9c655280d2c11277837c834b: Waiting for close lock at 1676307233661Disabling compacts and flushes for region at 1676307233661Disabling writes for close at 1676307233661Writing region close event to WAL at 1676307233661Closed at 1676307233661 2023-02-13 16:53:53,665 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 16:53:53,666 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307233666"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676307233666"}]},"ts":"1676307233666"} 2023-02-13 16:53:53,668 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 16:53:53,669 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 16:53:53,669 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307233669"}]},"ts":"1676307233669"} 2023-02-13 16:53:53,671 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLING in hbase:meta 2023-02-13 16:53:53,675 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 16:53:53,676 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 16:53:53,676 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 16:53:53,676 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 16:53:53,676 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 16:53:53,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=82053ffc9c655280d2c11277837c834b, ASSIGN}] 2023-02-13 16:53:53,680 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=82053ffc9c655280d2c11277837c834b, ASSIGN 2023-02-13 16:53:53,681 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=82053ffc9c655280d2c11277837c834b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39253,1676307231215; forceNewPlan=false, retain=false 2023-02-13 16:53:53,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 16:53:53,832 INFO [jenkins-hbase4:39061] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:53,833 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=82053ffc9c655280d2c11277837c834b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:53,833 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307233833"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307233833"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307233833"}]},"ts":"1676307233833"} 2023-02-13 16:53:53,835 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 82053ffc9c655280d2c11277837c834b, server=jenkins-hbase4.apache.org,39253,1676307231215}] 2023-02-13 16:53:53,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 16:53:53,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:53,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 82053ffc9c655280d2c11277837c834b, NAME => 'restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:53,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:53,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:53,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:53,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:53,994 INFO [StoreOpener-82053ffc9c655280d2c11277837c834b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:53,995 DEBUG [StoreOpener-82053ffc9c655280d2c11277837c834b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b/family 2023-02-13 16:53:53,995 DEBUG [StoreOpener-82053ffc9c655280d2c11277837c834b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b/family 2023-02-13 16:53:53,996 INFO [StoreOpener-82053ffc9c655280d2c11277837c834b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 82053ffc9c655280d2c11277837c834b columnFamilyName family 2023-02-13 16:53:53,996 INFO [StoreOpener-82053ffc9c655280d2c11277837c834b-1] regionserver.HStore(310): Store=82053ffc9c655280d2c11277837c834b/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:53,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:53,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:54,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:54,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 16:53:54,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 82053ffc9c655280d2c11277837c834b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62251733, jitterRate=-0.07237689197063446}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:54,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 82053ffc9c655280d2c11277837c834b: Running coprocessor pre-open hook at 1676307233992Writing region info on filesystem at 1676307233992Initializing all the Stores at 1676307233993 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307233993Cleaning up temporary data from old regions at 1676307233998 (+5 ms)Cleaning up detritus from prior splits at 1676307233999 (+1 ms)Running coprocessor post-open hooks at 1676307234004 (+5 ms)Region opened successfully at 1676307234004 2023-02-13 16:53:54,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b., pid=17, masterSystemTime=1676307233988 2023-02-13 16:53:54,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:54,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:54,008 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=82053ffc9c655280d2c11277837c834b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:54,008 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307234008"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307234008"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307234008"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307234008"}]},"ts":"1676307234008"} 2023-02-13 16:53:54,012 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-13 16:53:54,012 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 82053ffc9c655280d2c11277837c834b, server=jenkins-hbase4.apache.org,39253,1676307231215 in 175 msec 2023-02-13 16:53:54,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-13 16:53:54,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=82053ffc9c655280d2c11277837c834b, ASSIGN in 336 msec 2023-02-13 16:53:54,016 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 16:53:54,016 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"restartTableThree","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676307234016"}]},"ts":"1676307234016"} 2023-02-13 16:53:54,018 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=restartTableThree, state=ENABLED in hbase:meta 2023-02-13 16:53:54,020 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=restartTableThree execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 16:53:54,022 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree in 395 msec 2023-02-13 16:53:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39061] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-02-13 16:53:54,234 INFO [Listener at localhost/45077] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:restartTableThree, procId: 15 completed 2023-02-13 16:53:54,235 DEBUG [Listener at localhost/45077] hbase.HBaseTestingUtility(3430): Waiting until all regions of table restartTableThree get assigned. Timeout = 60000ms 2023-02-13 16:53:54,235 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:54,238 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(3484): All regions for table restartTableThree assigned to meta. Checking AM states. 2023-02-13 16:53:54,239 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:54,239 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(3504): All regions for table restartTableThree assigned. 2023-02-13 16:53:54,239 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:54,243 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:54,246 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:54,250 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:54,250 INFO [Listener at localhost/45077] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 16:53:54,253 INFO [Listener at localhost/45077] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 16:53:54,253 INFO [Listener at localhost/45077] master.TestRetainAssignmentOnRestart(88): Shutting down HBase cluster 2023-02-13 16:53:54,253 INFO [Listener at localhost/45077] hbase.MiniHBaseCluster(631): Stopping Thread[M:0;jenkins-hbase4:39061,5,FailOnTimeoutGroup] 2023-02-13 16:53:54,253 INFO [Listener at localhost/45077] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39061,1676307231115' ***** 2023-02-13 16:53:54,253 INFO [Listener at localhost/45077] regionserver.HRegionServer(2310): STOPPED: Stopping master 0 2023-02-13 16:53:54,255 DEBUG [M:0;jenkins-hbase4:39061] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c6e11e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:54,255 INFO [M:0;jenkins-hbase4:39061] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39061,1676307231115 2023-02-13 16:53:54,255 DEBUG [M:0;jenkins-hbase4:39061] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65c1ac1e to 127.0.0.1:61694 2023-02-13 16:53:54,255 DEBUG [M:0;jenkins-hbase4:39061] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:54,255 INFO [M:0;jenkins-hbase4:39061] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39061,1676307231115; all regions closed. 2023-02-13 16:53:54,256 DEBUG [M:0;jenkins-hbase4:39061] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:54,256 DEBUG [M:0;jenkins-hbase4:39061] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 16:53:54,256 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 16:53:54,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307231429] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307231429,5,FailOnTimeoutGroup] 2023-02-13 16:53:54,256 DEBUG [M:0;jenkins-hbase4:39061] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 16:53:54,258 INFO [M:0;jenkins-hbase4:39061] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 16:53:54,258 INFO [M:0;jenkins-hbase4:39061] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 16:53:54,257 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307231429] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307231429,5,FailOnTimeoutGroup] 2023-02-13 16:53:54,258 INFO [M:0;jenkins-hbase4:39061] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 16:53:54,259 DEBUG [M:0;jenkins-hbase4:39061] master.HMaster(1502): Stopping service threads 2023-02-13 16:53:54,259 INFO [M:0;jenkins-hbase4:39061] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 16:53:54,259 INFO [M:0;jenkins-hbase4:39061] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 16:53:54,259 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:54,259 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:54,259 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:54,260 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:54,260 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:54,260 DEBUG [Listener at localhost/45077] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 16:53:54,260 ERROR [M:0;jenkins-hbase4:39061] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-02-13 16:53:54,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-13 16:53:54,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:54,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:54,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:54,260 INFO [Listener at localhost/45077] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46119,1676307231186' ***** 2023-02-13 16:53:54,261 INFO [M:0;jenkins-hbase4:39061] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 16:53:54,261 INFO [Listener at localhost/45077] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 16:53:54,261 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 16:53:54,261 INFO [Listener at localhost/45077] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39253,1676307231215' ***** 2023-02-13 16:53:54,261 INFO [RS:0;jenkins-hbase4:46119] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:53:54,261 DEBUG [M:0;jenkins-hbase4:39061] zookeeper.ZKUtil(398): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 16:53:54,261 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:53:54,261 WARN [M:0;jenkins-hbase4:39061] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 16:53:54,262 INFO [M:0;jenkins-hbase4:39061] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 16:53:54,261 INFO [Listener at localhost/45077] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 16:53:54,262 INFO [M:0;jenkins-hbase4:39061] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 16:53:54,263 INFO [Listener at localhost/45077] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39009,1676307231248' ***** 2023-02-13 16:53:54,261 INFO [RS:0;jenkins-hbase4:46119] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:53:54,263 INFO [Listener at localhost/45077] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 16:53:54,263 DEBUG [M:0;jenkins-hbase4:39061] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 16:53:54,263 INFO [RS:1;jenkins-hbase4:39253] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:53:54,263 INFO [M:0;jenkins-hbase4:39061] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:54,263 INFO [RS:2;jenkins-hbase4:39009] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:53:54,263 INFO [RS:0;jenkins-hbase4:46119] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:53:54,264 INFO [RS:2;jenkins-hbase4:39009] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:53:54,264 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(3304): Received CLOSE for 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:54,264 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:53:54,264 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:53:54,264 INFO [RS:1;jenkins-hbase4:39253] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:53:54,264 DEBUG [M:0;jenkins-hbase4:39061] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:54,265 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:54,264 INFO [RS:1;jenkins-hbase4:39253] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:53:54,264 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:54,265 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(3304): Received CLOSE for b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:54,264 INFO [RS:2;jenkins-hbase4:39009] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:53:54,265 DEBUG [RS:0;jenkins-hbase4:46119] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2922c961 to 127.0.0.1:61694 2023-02-13 16:53:54,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 745e4bc8b1326452bfdd5dbda9c99aeb, disabling compactions & flushes 2023-02-13 16:53:54,265 DEBUG [M:0;jenkins-hbase4:39061] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 16:53:54,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b1fad04018b6f3dcf4291cca06397285, disabling compactions & flushes 2023-02-13 16:53:54,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:54,266 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(3304): Received CLOSE for 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:54,266 DEBUG [RS:0;jenkins-hbase4:46119] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:54,267 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:54,266 INFO [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(3304): Received CLOSE for c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:54,268 DEBUG [RS:1;jenkins-hbase4:39253] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x513051d3 to 127.0.0.1:61694 2023-02-13 16:53:54,268 INFO [RS:0;jenkins-hbase4:46119] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:53:54,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:54,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:54,267 DEBUG [M:0;jenkins-hbase4:39061] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:54,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:54,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. after waiting 0 ms 2023-02-13 16:53:54,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:54,269 INFO [M:0;jenkins-hbase4:39061] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.19 KB heapSize=76.14 KB 2023-02-13 16:53:54,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c49f69a5b49e9a6b371b23ed03123cca, disabling compactions & flushes 2023-02-13 16:53:54,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. after waiting 0 ms 2023-02-13 16:53:54,268 INFO [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:54,268 INFO [RS:0;jenkins-hbase4:46119] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:53:54,269 INFO [RS:0;jenkins-hbase4:46119] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:53:54,268 DEBUG [RS:1;jenkins-hbase4:39253] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:54,270 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 16:53:54,270 DEBUG [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1479): Online Regions={b1fad04018b6f3dcf4291cca06397285=restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285., 82053ffc9c655280d2c11277837c834b=restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.} 2023-02-13 16:53:54,270 DEBUG [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1505): Waiting on 82053ffc9c655280d2c11277837c834b, b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:54,270 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 16:53:54,269 DEBUG [RS:2;jenkins-hbase4:39009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x595aaab6 to 127.0.0.1:61694 2023-02-13 16:53:54,271 DEBUG [RS:2;jenkins-hbase4:39009] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:54,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:54,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:54,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 745e4bc8b1326452bfdd5dbda9c99aeb 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 16:53:54,272 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 16:53:54,272 INFO [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 16:53:54,271 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 16:53:54,272 DEBUG [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1479): Online Regions={c49f69a5b49e9a6b371b23ed03123cca=restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.} 2023-02-13 16:53:54,272 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 16:53:54,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:54,273 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 16:53:54,273 DEBUG [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1505): Waiting on c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:54,273 DEBUG [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 745e4bc8b1326452bfdd5dbda9c99aeb=hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.} 2023-02-13 16:53:54,273 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 16:53:54,273 DEBUG [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1505): Waiting on 1588230740, 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:54,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. after waiting 0 ms 2023-02-13 16:53:54,273 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 16:53:54,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:54,273 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=5.14 KB heapSize=9.42 KB 2023-02-13 16:53:54,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 16:53:54,294 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:54,294 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b1fad04018b6f3dcf4291cca06397285: Waiting for close lock at 1676307234267Running coprocessor pre-close hooks at 1676307234267Disabling compacts and flushes for region at 1676307234267Disabling writes for close at 1676307234268 (+1 ms)Writing region close event to WAL at 1676307234269 (+1 ms)Running coprocessor post-close hooks at 1676307234293 (+24 ms)Closed at 1676307234294 (+1 ms) 2023-02-13 16:53:54,294 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:54,294 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 16:53:54,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 82053ffc9c655280d2c11277837c834b, disabling compactions & flushes 2023-02-13 16:53:54,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:54,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c49f69a5b49e9a6b371b23ed03123cca: Waiting for close lock at 1676307234268Running coprocessor pre-close hooks at 1676307234268Disabling compacts and flushes for region at 1676307234268Disabling writes for close at 1676307234273 (+5 ms)Writing region close event to WAL at 1676307234274 (+1 ms)Running coprocessor post-close hooks at 1676307234295 (+21 ms)Closed at 1676307234297 (+2 ms) 2023-02-13 16:53:54,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:54,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:54,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. after waiting 0 ms 2023-02-13 16:53:54,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:54,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:54,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 16:53:54,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:54,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 82053ffc9c655280d2c11277837c834b: Waiting for close lock at 1676307234294Running coprocessor pre-close hooks at 1676307234294Disabling compacts and flushes for region at 1676307234294Disabling writes for close at 1676307234297 (+3 ms)Writing region close event to WAL at 1676307234302 (+5 ms)Running coprocessor post-close hooks at 1676307234322 (+20 ms)Closed at 1676307234322 2023-02-13 16:53:54,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:54,328 INFO [M:0;jenkins-hbase4:39061] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.19 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ce6f6d831a204dada8d75104d47f5ed7 2023-02-13 16:53:54,336 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/.tmp/info/794b2c13eb0644c099a6891718649f3f 2023-02-13 16:53:54,340 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.77 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/.tmp/info/f883d91573954b5aa8708e17bd64ea21 2023-02-13 16:53:54,342 DEBUG [M:0;jenkins-hbase4:39061] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ce6f6d831a204dada8d75104d47f5ed7 as hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ce6f6d831a204dada8d75104d47f5ed7 2023-02-13 16:53:54,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/.tmp/info/794b2c13eb0644c099a6891718649f3f as hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/info/794b2c13eb0644c099a6891718649f3f 2023-02-13 16:53:54,351 INFO [M:0;jenkins-hbase4:39061] regionserver.HStore(1080): Added hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ce6f6d831a204dada8d75104d47f5ed7, entries=17, sequenceid=141, filesize=8.5 K 2023-02-13 16:53:54,353 INFO [M:0;jenkins-hbase4:39061] regionserver.HRegion(2947): Finished flush of dataSize ~64.19 KB/65731, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 85ms, sequenceid=141, compaction requested=false 2023-02-13 16:53:54,355 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:54,355 INFO [M:0;jenkins-hbase4:39061] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:53:54,356 DEBUG [M:0;jenkins-hbase4:39061] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676307234263Disabling compacts and flushes for region at 1676307234263Disabling writes for close at 1676307234267 (+4 ms)Obtaining lock to block concurrent updates at 1676307234269 (+2 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676307234269Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=65731, getHeapSize=77952, getOffHeapSize=0, getCellsCount=158 at 1676307234269Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676307234270 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676307234271 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676307234289 (+18 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676307234289Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676307234343 (+54 ms)Finished flush of dataSize ~64.19 KB/65731, heapSize ~76.13 KB/77952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 85ms, sequenceid=141, compaction requested=false at 1676307234353 (+10 ms)Writing region close event to WAL at 1676307234355 (+2 ms)Closed at 1676307234355 2023-02-13 16:53:54,363 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:54,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/info/794b2c13eb0644c099a6891718649f3f, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 16:53:54,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 745e4bc8b1326452bfdd5dbda9c99aeb in 99ms, sequenceid=6, compaction requested=false 2023-02-13 16:53:54,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 16:53:54,375 INFO [M:0;jenkins-hbase4:39061] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 16:53:54,375 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:54,377 INFO [M:0;jenkins-hbase4:39061] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39061 2023-02-13 16:53:54,381 DEBUG [M:0;jenkins-hbase4:39061] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39061,1676307231115 already deleted, retry=false 2023-02-13 16:53:54,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 16:53:54,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:54,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 745e4bc8b1326452bfdd5dbda9c99aeb: Waiting for close lock at 1676307234265Running coprocessor pre-close hooks at 1676307234265Disabling compacts and flushes for region at 1676307234265Disabling writes for close at 1676307234269 (+4 ms)Obtaining lock to block concurrent updates at 1676307234272 (+3 ms)Preparing flush snapshotting stores in 745e4bc8b1326452bfdd5dbda9c99aeb at 1676307234272Finished memstore snapshotting hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676307234272Flushing stores of hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. at 1676307234273 (+1 ms)Flushing 745e4bc8b1326452bfdd5dbda9c99aeb/info: creating writer at 1676307234273Flushing 745e4bc8b1326452bfdd5dbda9c99aeb/info: appending metadata at 1676307234286 (+13 ms)Flushing 745e4bc8b1326452bfdd5dbda9c99aeb/info: closing flushed file at 1676307234286Flushing 745e4bc8b1326452bfdd5dbda9c99aeb/info: reopening flushed file at 1676307234351 (+65 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 745e4bc8b1326452bfdd5dbda9c99aeb in 99ms, sequenceid=6, compaction requested=false at 1676307234371 (+20 ms)Writing region close event to WAL at 1676307234378 (+7 ms)Running coprocessor post-close hooks at 1676307234387 (+9 ms)Closed at 1676307234388 (+1 ms) 2023-02-13 16:53:54,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:54,390 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=380 B at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/.tmp/table/dffd9fac4743453bb3301484d2d73476 2023-02-13 16:53:54,397 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/.tmp/info/f883d91573954b5aa8708e17bd64ea21 as hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info/f883d91573954b5aa8708e17bd64ea21 2023-02-13 16:53:54,403 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info/f883d91573954b5aa8708e17bd64ea21, entries=40, sequenceid=24, filesize=9.5 K 2023-02-13 16:53:54,404 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/.tmp/table/dffd9fac4743453bb3301484d2d73476 as hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/table/dffd9fac4743453bb3301484d2d73476 2023-02-13 16:53:54,411 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/table/dffd9fac4743453bb3301484d2d73476, entries=8, sequenceid=24, filesize=5.0 K 2023-02-13 16:53:54,412 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 138ms, sequenceid=24, compaction requested=false 2023-02-13 16:53:54,420 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=1 2023-02-13 16:53:54,421 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 16:53:54,421 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 16:53:54,421 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676307234272Running coprocessor pre-close hooks at 1676307234272Disabling compacts and flushes for region at 1676307234272Disabling writes for close at 1676307234273 (+1 ms)Obtaining lock to block concurrent updates at 1676307234273Preparing flush snapshotting stores in 1588230740 at 1676307234273Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=5262, getHeapSize=9600, getOffHeapSize=0, getCellsCount=48 at 1676307234274 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676307234276 (+2 ms)Flushing 1588230740/info: creating writer at 1676307234276Flushing 1588230740/info: appending metadata at 1676307234285 (+9 ms)Flushing 1588230740/info: closing flushed file at 1676307234285Flushing 1588230740/table: creating writer at 1676307234349 (+64 ms)Flushing 1588230740/table: appending metadata at 1676307234356 (+7 ms)Flushing 1588230740/table: closing flushed file at 1676307234356Flushing 1588230740/info: reopening flushed file at 1676307234397 (+41 ms)Flushing 1588230740/table: reopening flushed file at 1676307234405 (+8 ms)Finished flush of dataSize ~5.14 KB/5262, heapSize ~9.14 KB/9360, currentSize=0 B/0 for 1588230740 in 138ms, sequenceid=24, compaction requested=false at 1676307234412 (+7 ms)Writing region close event to WAL at 1676307234416 (+4 ms)Running coprocessor post-close hooks at 1676307234421 (+5 ms)Closed at 1676307234421 2023-02-13 16:53:54,421 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 16:53:54,470 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39253,1676307231215; all regions closed. 2023-02-13 16:53:54,473 INFO [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39009,1676307231248; all regions closed. 2023-02-13 16:53:54,473 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46119,1676307231186; all regions closed. 2023-02-13 16:53:54,481 DEBUG [RS:1;jenkins-hbase4:39253] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs 2023-02-13 16:53:54,482 INFO [RS:1;jenkins-hbase4:39253] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39253%2C1676307231215:(num 1676307231694) 2023-02-13 16:53:54,482 DEBUG [RS:1;jenkins-hbase4:39253] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:54,482 INFO [RS:1;jenkins-hbase4:39253] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:54,482 INFO [RS:1;jenkins-hbase4:39253] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 16:53:54,482 INFO [RS:1;jenkins-hbase4:39253] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:53:54,482 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:54,482 INFO [RS:1;jenkins-hbase4:39253] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:53:54,482 INFO [RS:1;jenkins-hbase4:39253] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:53:54,484 DEBUG [RS:0;jenkins-hbase4:46119] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs 2023-02-13 16:53:54,484 INFO [RS:0;jenkins-hbase4:46119] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46119%2C1676307231186.meta:.meta(num 1676307231804) 2023-02-13 16:53:54,486 INFO [RS:1;jenkins-hbase4:39253] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39253 2023-02-13 16:53:54,491 DEBUG [RS:2;jenkins-hbase4:39009] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs 2023-02-13 16:53:54,491 INFO [RS:2;jenkins-hbase4:39009] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39009%2C1676307231248:(num 1676307231691) 2023-02-13 16:53:54,491 DEBUG [RS:2;jenkins-hbase4:39009] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:54,491 INFO [RS:2;jenkins-hbase4:39009] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:54,491 INFO [RS:2;jenkins-hbase4:39009] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 16:53:54,491 INFO [RS:2;jenkins-hbase4:39009] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:53:54,491 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:54,492 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:54,492 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:54,492 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:54,492 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:54,491 INFO [RS:2;jenkins-hbase4:39009] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:53:54,492 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:54,492 INFO [RS:2;jenkins-hbase4:39009] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:53:54,492 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@281fcaff rejected from java.util.concurrent.ThreadPoolExecutor@2c204b24[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,493 INFO [RS:2;jenkins-hbase4:39009] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39009 2023-02-13 16:53:54,493 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:54,493 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@e9f9734 rejected from java.util.concurrent.ThreadPoolExecutor@2c204b24[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,498 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:54,498 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:54,499 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@14f7a945 rejected from java.util.concurrent.ThreadPoolExecutor@10c83f03[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,500 DEBUG [RS:0;jenkins-hbase4:46119] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs 2023-02-13 16:53:54,500 INFO [RS:0;jenkins-hbase4:46119] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46119%2C1676307231186:(num 1676307231691) 2023-02-13 16:53:54,500 DEBUG [RS:0;jenkins-hbase4:46119] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:53:54,500 INFO [RS:0;jenkins-hbase4:46119] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:53:54,501 INFO [RS:0;jenkins-hbase4:46119] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 16:53:54,501 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:53:54,501 INFO [RS:0;jenkins-hbase4:46119] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46119 2023-02-13 16:53:54,503 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:54,503 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1a04356f rejected from java.util.concurrent.ThreadPoolExecutor@1e3bb2f7[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,764 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:54,764 INFO [RS:0;jenkins-hbase4:46119] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46119,1676307231186; zookeeper connection closed. 2023-02-13 16:53:54,764 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@51bcda55 rejected from java.util.concurrent.ThreadPoolExecutor@1e3bb2f7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,765 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@63445814] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@63445814 2023-02-13 16:53:54,765 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b350001, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:54,765 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@294799f8 rejected from java.util.concurrent.ThreadPoolExecutor@1e3bb2f7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,864 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:54,864 INFO [RS:2;jenkins-hbase4:39009] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39009,1676307231248; zookeeper connection closed. 2023-02-13 16:53:54,864 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4dc1e678 rejected from java.util.concurrent.ThreadPoolExecutor@10c83f03[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,865 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@14e9b935] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@14e9b935 2023-02-13 16:53:54,865 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b350003, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:54,866 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@35b4c8aa rejected from java.util.concurrent.ThreadPoolExecutor@10c83f03[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,964 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:54,964 INFO [RS:1;jenkins-hbase4:39253] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39253,1676307231215; zookeeper connection closed. 2023-02-13 16:53:54,965 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7656e565 rejected from java.util.concurrent.ThreadPoolExecutor@2c204b24[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,965 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b350002, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:54,965 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7bd93fa9 rejected from java.util.concurrent.ThreadPoolExecutor@2c204b24[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:54,965 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@453eaeac] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@453eaeac 2023-02-13 16:53:54,967 INFO [Listener at localhost/45077] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-13 16:53:55,065 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:55,065 INFO [M:0;jenkins-hbase4:39061] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39061,1676307231115; zookeeper connection closed. 2023-02-13 16:53:55,065 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@187182c5 rejected from java.util.concurrent.ThreadPoolExecutor@1c4e1513[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:55,066 INFO [Listener at localhost/45077] master.TestRetainAssignmentOnRestart(93): Sleeping a bit 2023-02-13 16:53:55,066 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:39061-0x1004e9a5b350000, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:53:55,066 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7cacb0d2 rejected from java.util.concurrent.ThreadPoolExecutor@1c4e1513[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:53:57,066 INFO [Listener at localhost/45077] master.TestRetainAssignmentOnRestart(96): Starting cluster the second time with the same ports 2023-02-13 16:53:57,079 INFO [Listener at localhost/45077] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-13 16:53:57,079 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,079 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,079 INFO [Listener at localhost/45077] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:57,079 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,079 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:57,079 INFO [Listener at localhost/45077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:57,080 INFO [Listener at localhost/45077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42765 2023-02-13 16:53:57,081 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:57,082 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:57,083 INFO [Listener at localhost/45077] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42765 connecting to ZooKeeper ensemble=127.0.0.1:61694 2023-02-13 16:53:57,088 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:427650x0, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:57,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:42765-0x1004e9a5b350009 connected 2023-02-13 16:53:57,093 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:53:57,093 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(162): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 16:53:57,094 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:57,094 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42765 2023-02-13 16:53:57,094 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42765 2023-02-13 16:53:57,099 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42765 2023-02-13 16:53:57,099 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42765 2023-02-13 16:53:57,099 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42765 2023-02-13 16:53:57,100 INFO [Listener at localhost/45077] master.HMaster(439): hbase.rootdir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76, hbase.cluster.distributed=false 2023-02-13 16:53:57,100 DEBUG [pool-505-thread-1] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: INIT 2023-02-13 16:53:57,101 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,103 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 16:53:57,103 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,105 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 16:53:57,105 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:53:57,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:57,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:57,106 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,42765,1676307237078 from backup master directory 2023-02-13 16:53:57,106 INFO [M:1;jenkins-hbase4:42765] regionserver.HRegionServer(952): ClusterId : 1d915b4a-57b4-498c-b754-ef7b45ac1780 2023-02-13 16:53:57,108 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,108 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:57,108 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 16:53:57,108 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,116 INFO [Listener at localhost/45077] client.ConnectionUtils(127): regionserver/jenkins-hbase4:46119 server-side Connection retries=45 2023-02-13 16:53:57,116 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,116 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,116 INFO [Listener at localhost/45077] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:57,116 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,116 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:57,116 INFO [Listener at localhost/45077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:57,117 INFO [Listener at localhost/45077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46119 2023-02-13 16:53:57,117 INFO [Listener at localhost/45077] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:57,118 DEBUG [Listener at localhost/45077] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:57,119 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:57,121 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:57,122 INFO [Listener at localhost/45077] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46119 connecting to ZooKeeper ensemble=127.0.0.1:61694 2023-02-13 16:53:57,126 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:461190x0, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:57,127 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(162): regionserver:461190x0, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:57,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46119-0x1004e9a5b35000a connected 2023-02-13 16:53:57,128 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 16:53:57,129 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:57,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:57,132 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46119 2023-02-13 16:53:57,132 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46119 2023-02-13 16:53:57,132 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46119 2023-02-13 16:53:57,133 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46119 2023-02-13 16:53:57,133 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46119 2023-02-13 16:53:57,136 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(952): ClusterId : 1d915b4a-57b4-498c-b754-ef7b45ac1780 2023-02-13 16:53:57,136 DEBUG [RS:3;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:57,139 DEBUG [RS:3;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:57,139 DEBUG [RS:3;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:57,141 DEBUG [RS:3;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:57,142 DEBUG [RS:3;jenkins-hbase4:46119] zookeeper.ReadOnlyZKClient(139): Connect 0x7d8d4df1 to 127.0.0.1:61694 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:57,147 DEBUG [RS:3;jenkins-hbase4:46119] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@734abc30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:57,147 DEBUG [RS:3;jenkins-hbase4:46119] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77752681, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:57,157 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:3;jenkins-hbase4:46119 2023-02-13 16:53:57,157 INFO [RS:3;jenkins-hbase4:46119] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:57,158 INFO [RS:3;jenkins-hbase4:46119] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:57,158 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:57,158 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42765,1676307237078 with isa=jenkins-hbase4.apache.org/172.31.14.131:46119, startcode=1676307237115 2023-02-13 16:53:57,158 DEBUG [RS:3;jenkins-hbase4:46119] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:57,161 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45795, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.9 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:57,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:57,162 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 16:53:57,162 WARN [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 16:53:57,167 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x37ca281d to 127.0.0.1:61694 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:57,171 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2797cfb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:57,171 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 16:53:57,171 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 16:53:57,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:57,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,39061,1676307231115 to hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,39061,1676307231115-dead as it is dead 2023-02-13 16:53:57,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,39061,1676307231115-dead/jenkins-hbase4.apache.org%2C39061%2C1676307231115.1676307231344 2023-02-13 16:53:57,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,39061,1676307231115-dead/jenkins-hbase4.apache.org%2C39061%2C1676307231115.1676307231344 after 0ms 2023-02-13 16:53:57,179 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,39061,1676307231115-dead/jenkins-hbase4.apache.org%2C39061%2C1676307231115.1676307231344 to hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39061%2C1676307231115.1676307231344 2023-02-13 16:53:57,179 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,39061,1676307231115-dead 2023-02-13 16:53:57,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42765%2C1676307237078, suffix=, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,42765,1676307237078, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/oldWALs, maxLogs=10 2023-02-13 16:53:57,197 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:57,198 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:57,198 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:57,201 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/WALs/jenkins-hbase4.apache.org,42765,1676307237078/jenkins-hbase4.apache.org%2C42765%2C1676307237078.1676307237182 2023-02-13 16:53:57,201 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK], DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK], DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK]] 2023-02-13 16:53:57,201 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:57,201 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:57,201 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:57,201 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:57,202 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:57,203 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 16:53:57,204 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 16:53:57,211 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(539): loaded hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ce6f6d831a204dada8d75104d47f5ed7 2023-02-13 16:53:57,211 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:57,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5178): Found 1 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-02-13 16:53:57,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5275): Replaying edits from hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39061%2C1676307231115.1676307231344 2023-02-13 16:53:57,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5463): Applied 0, skipped 160, firstSequenceIdInLog=3, maxSequenceIdInLog=143, path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39061%2C1676307231115.1676307231344 2023-02-13 16:53:57,219 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5085): Deleted recovered.edits file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C39061%2C1676307231115.1676307231344 2023-02-13 16:53:57,223 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 16:53:57,225 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/143.seqid, newMaxSeqId=143, maxSeqId=1 2023-02-13 16:53:57,226 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=144; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69109946, jitterRate=0.029818445444107056}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:57,226 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676307237201Initializing all the Stores at 1676307237202 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307237202Cleaning up temporary data from old regions at 1676307237220 (+18 ms)Cleaning up detritus from prior splits at 1676307237220Region opened successfully at 1676307237226 (+6 ms) 2023-02-13 16:53:57,227 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 16:53:57,228 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 16:53:57,228 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 16:53:57,228 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 16:53:57,228 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 16:53:57,233 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-13 16:53:57,233 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-13 16:53:57,234 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-13 16:53:57,234 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-13 16:53:57,234 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=restartTableOne 2023-02-13 16:53:57,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; CreateTableProcedure table=restartTableTwo 2023-02-13 16:53:57,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=15, state=SUCCESS; CreateTableProcedure table=restartTableThree 2023-02-13 16:53:57,235 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-13 16:53:57,236 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 16:53:57,241 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-13 16:53:57,241 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,46119,1676307231186, table=hbase:meta, region=1588230740 2023-02-13 16:53:57,243 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 3 possibly 'live' servers, and 0 'splitting'. 2023-02-13 16:53:57,244 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39253,1676307231215 already deleted, retry=false 2023-02-13 16:53:57,244 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,39253,1676307231215 on jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,245 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,39253,1676307231215, splitWal=true, meta=false 2023-02-13 16:53:57,245 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=18 for jenkins-hbase4.apache.org,39253,1676307231215 (carryingMeta=false) jenkins-hbase4.apache.org,39253,1676307231215/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2160436b[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 16:53:57,247 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39009,1676307231248 already deleted, retry=false 2023-02-13 16:53:57,247 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,39009,1676307231248 on jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,249 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,39009,1676307231248, splitWal=true, meta=false 2023-02-13 16:53:57,249 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=19 for jenkins-hbase4.apache.org,39009,1676307231248 (carryingMeta=false) jenkins-hbase4.apache.org,39009,1676307231248/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@450e0a4c[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 16:53:57,250 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46119,1676307231186 already deleted, retry=false 2023-02-13 16:53:57,250 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,46119,1676307231186 on jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,255 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=20, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,46119,1676307231186, splitWal=true, meta=true 2023-02-13 16:53:57,255 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1736): Scheduled ServerCrashProcedure pid=20 for jenkins-hbase4.apache.org,46119,1676307231186 (carryingMeta=true) jenkins-hbase4.apache.org,46119,1676307231186/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@26a2cfd[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-02-13 16:53:57,263 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42765,1676307237078 with isa=jenkins-hbase4.apache.org/172.31.14.131:46119, startcode=1676307237115 2023-02-13 16:53:57,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:57,264 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 16:53:57,264 WARN [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 200 ms and then retrying. 2023-02-13 16:53:57,267 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 16:53:57,267 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 16:53:57,267 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/balancer 2023-02-13 16:53:57,268 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 16:53:57,268 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 16:53:57,269 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 16:53:57,269 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 16:53:57,270 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 16:53:57,270 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,42765,1676307237078, sessionid=0x1004e9a5b350009, setting cluster-up flag (Was=true) 2023-02-13 16:53:57,272 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 16:53:57,272 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,275 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 16:53:57,275 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:53:57,276 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/.hbase-snapshot/.tmp 2023-02-13 16:53:57,280 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:57,281 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:57,281 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:57,281 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 16:53:57,281 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 16:53:57,281 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,281 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:57,281 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,285 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676307267284 2023-02-13 16:53:57,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 16:53:57,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 16:53:57,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 16:53:57,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 16:53:57,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 16:53:57,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 16:53:57,288 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,291 DEBUG [PEWorker-1] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46119,1676307231186; numProcessing=1 2023-02-13 16:53:57,291 DEBUG [PEWorker-4] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39009,1676307231248; numProcessing=2 2023-02-13 16:53:57,291 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=20, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46119,1676307231186, splitWal=true, meta=true 2023-02-13 16:53:57,291 DEBUG [PEWorker-2] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39253,1676307231215; numProcessing=3 2023-02-13 16:53:57,291 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 16:53:57,291 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=19, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39009,1676307231248, splitWal=true, meta=false 2023-02-13 16:53:57,292 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 16:53:57,291 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=18, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39253,1676307231215, splitWal=true, meta=false 2023-02-13 16:53:57,292 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 16:53:57,293 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=20, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46119,1676307231186, splitWal=true, meta=true, isMeta: true 2023-02-13 16:53:57,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 16:53:57,297 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 16:53:57,298 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186-splitting 2023-02-13 16:53:57,299 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186-splitting dir is empty, no logs to split. 2023-02-13 16:53:57,299 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,46119,1676307231186 WAL count=0, meta=true 2023-02-13 16:53:57,303 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307237297,5,FailOnTimeoutGroup] 2023-02-13 16:53:57,303 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307237303,5,FailOnTimeoutGroup] 2023-02-13 16:53:57,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,304 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 16:53:57,304 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,304 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,304 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676307237103, completionTime=-1 2023-02-13 16:53:57,304 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=0; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 16:53:57,305 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186-splitting dir is empty, no logs to split. 2023-02-13 16:53:57,305 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,46119,1676307231186 WAL count=0, meta=true 2023-02-13 16:53:57,305 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,46119,1676307231186 WAL splitting is done? wals=0, meta=true 2023-02-13 16:53:57,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 16:53:57,313 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 16:53:57,315 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,46119,1676307231186; forceNewPlan=false, retain=false 2023-02-13 16:53:57,465 WARN [jenkins-hbase4:42765] assignment.AssignmentManager(2157): No servers available; cannot place 1 unassigned regions. 2023-02-13 16:53:57,465 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42765,1676307237078 with isa=jenkins-hbase4.apache.org/172.31.14.131:46119, startcode=1676307237115 2023-02-13 16:53:57,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,467 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76 2023-02-13 16:53:57,467 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37465 2023-02-13 16:53:57,467 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:57,470 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:57,470 DEBUG [RS:3;jenkins-hbase4:46119] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,471 WARN [RS:3;jenkins-hbase4:46119] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:57,471 INFO [RS:3;jenkins-hbase4:46119] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:57,471 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,471 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46119,1676307237115] 2023-02-13 16:53:57,476 DEBUG [RS:3;jenkins-hbase4:46119] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,476 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:57,477 INFO [RS:3;jenkins-hbase4:46119] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:57,478 INFO [RS:3;jenkins-hbase4:46119] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:57,480 INFO [RS:3;jenkins-hbase4:46119] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:57,480 INFO [RS:3;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,480 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:57,482 INFO [RS:3;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,482 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:46119, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,482 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:46119, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,482 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,482 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,482 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:46119, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,482 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:46119, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:57,482 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:46119, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,482 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:46119, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,482 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:46119, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,483 DEBUG [RS:3;jenkins-hbase4:46119] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:46119, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,487 INFO [RS:3;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,487 INFO [RS:3;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,487 INFO [RS:3;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,498 INFO [RS:3;jenkins-hbase4:46119] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:57,498 INFO [RS:3;jenkins-hbase4:46119] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46119,1676307237115-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,504 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=1; waited=200ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 16:53:57,509 INFO [RS:3;jenkins-hbase4:46119] regionserver.Replication(203): jenkins-hbase4.apache.org,46119,1676307237115 started 2023-02-13 16:53:57,509 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46119,1676307237115, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46119, sessionid=0x1004e9a5b35000a 2023-02-13 16:53:57,509 DEBUG [RS:3;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:57,509 DEBUG [RS:3;jenkins-hbase4:46119] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,509 DEBUG [RS:3;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46119,1676307237115' 2023-02-13 16:53:57,509 DEBUG [RS:3;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:57,509 DEBUG [RS:3;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:57,510 DEBUG [RS:3;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:57,510 DEBUG [RS:3;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:57,510 DEBUG [RS:3;jenkins-hbase4:46119] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,510 DEBUG [RS:3;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46119,1676307237115' 2023-02-13 16:53:57,510 DEBUG [RS:3;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:57,510 DEBUG [RS:3;jenkins-hbase4:46119] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:57,510 DEBUG [RS:3;jenkins-hbase4:46119] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:57,511 INFO [RS:3;jenkins-hbase4:46119] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:57,511 INFO [RS:3;jenkins-hbase4:46119] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:57,520 INFO [Listener at localhost/45077] client.ConnectionUtils(127): regionserver/jenkins-hbase4:39253 server-side Connection retries=45 2023-02-13 16:53:57,520 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,521 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,521 INFO [Listener at localhost/45077] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:57,521 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,521 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:57,521 INFO [Listener at localhost/45077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:57,522 INFO [Listener at localhost/45077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39253 2023-02-13 16:53:57,522 INFO [Listener at localhost/45077] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:57,523 DEBUG [Listener at localhost/45077] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:57,524 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:57,525 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:57,526 INFO [Listener at localhost/45077] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39253 connecting to ZooKeeper ensemble=127.0.0.1:61694 2023-02-13 16:53:57,530 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:392530x0, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:57,531 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(162): regionserver:392530x0, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:57,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39253-0x1004e9a5b35000d connected 2023-02-13 16:53:57,532 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 16:53:57,533 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:57,536 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39253 2023-02-13 16:53:57,536 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39253 2023-02-13 16:53:57,536 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39253 2023-02-13 16:53:57,537 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39253 2023-02-13 16:53:57,540 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39253 2023-02-13 16:53:57,543 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(952): ClusterId : 1d915b4a-57b4-498c-b754-ef7b45ac1780 2023-02-13 16:53:57,544 DEBUG [RS:4;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:57,546 DEBUG [RS:4;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:57,546 DEBUG [RS:4;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:57,548 DEBUG [RS:4;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:57,549 DEBUG [RS:4;jenkins-hbase4:39253] zookeeper.ReadOnlyZKClient(139): Connect 0x3defb754 to 127.0.0.1:61694 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:57,556 DEBUG [RS:4;jenkins-hbase4:39253] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33284e99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:57,556 DEBUG [RS:4;jenkins-hbase4:39253] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3717dc24, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:57,565 DEBUG [RS:4;jenkins-hbase4:39253] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:4;jenkins-hbase4:39253 2023-02-13 16:53:57,565 INFO [RS:4;jenkins-hbase4:39253] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:57,565 INFO [RS:4;jenkins-hbase4:39253] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:57,565 DEBUG [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:57,566 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42765,1676307237078 with isa=jenkins-hbase4.apache.org/172.31.14.131:39253, startcode=1676307237520 2023-02-13 16:53:57,566 DEBUG [RS:4;jenkins-hbase4:39253] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:57,570 INFO [RS-EventLoopGroup-13-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35961, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.10 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:57,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,571 DEBUG [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76 2023-02-13 16:53:57,571 DEBUG [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37465 2023-02-13 16:53:57,571 DEBUG [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:57,573 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:57,573 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:57,574 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39253,1676307237520] 2023-02-13 16:53:57,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,574 DEBUG [RS:4;jenkins-hbase4:39253] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,574 WARN [RS:4;jenkins-hbase4:39253] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:57,574 INFO [RS:4;jenkins-hbase4:39253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:57,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,574 DEBUG [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,580 DEBUG [RS:4;jenkins-hbase4:39253] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,581 DEBUG [RS:4;jenkins-hbase4:39253] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,581 DEBUG [RS:4;jenkins-hbase4:39253] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:57,582 INFO [RS:4;jenkins-hbase4:39253] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:57,584 INFO [RS:4;jenkins-hbase4:39253] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:57,586 INFO [RS:4;jenkins-hbase4:39253] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:57,586 INFO [RS:4;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,587 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:57,588 INFO [RS:4;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:39253, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:39253, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:39253, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:39253, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:39253, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:39253, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:39253, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:39253, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,589 DEBUG [RS:4;jenkins-hbase4:39253] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:39253, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,590 INFO [RS:4;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,590 INFO [RS:4;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,590 INFO [RS:4;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(801): Waiting on regionserver count=2; waited=300ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2023-02-13 16:53:57,606 INFO [RS:4;jenkins-hbase4:39253] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:57,606 INFO [RS:4;jenkins-hbase4:39253] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39253,1676307237520-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,613 INFO [RS:3;jenkins-hbase4:46119] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46119%2C1676307237115, suffix=, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307237115, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs, maxLogs=32 2023-02-13 16:53:57,619 INFO [RS:4;jenkins-hbase4:39253] regionserver.Replication(203): jenkins-hbase4.apache.org,39253,1676307237520 started 2023-02-13 16:53:57,619 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39253,1676307237520, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39253, sessionid=0x1004e9a5b35000d 2023-02-13 16:53:57,619 DEBUG [RS:4;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:57,619 DEBUG [RS:4;jenkins-hbase4:39253] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,619 DEBUG [RS:4;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39253,1676307237520' 2023-02-13 16:53:57,619 DEBUG [RS:4;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:57,620 DEBUG [RS:4;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:57,620 DEBUG [RS:4;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:57,620 DEBUG [RS:4;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:57,620 DEBUG [RS:4;jenkins-hbase4:39253] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,620 DEBUG [RS:4;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39253,1676307237520' 2023-02-13 16:53:57,620 DEBUG [RS:4;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:57,621 DEBUG [RS:4;jenkins-hbase4:39253] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:57,621 DEBUG [RS:4;jenkins-hbase4:39253] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:57,621 INFO [RS:4;jenkins-hbase4:39253] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:57,621 INFO [RS:4;jenkins-hbase4:39253] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:57,633 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:57,633 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:57,633 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:57,641 INFO [RS:3;jenkins-hbase4:46119] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307237115/jenkins-hbase4.apache.org%2C46119%2C1676307237115.1676307237613 2023-02-13 16:53:57,641 DEBUG [RS:3;jenkins-hbase4:46119] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK], DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK], DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK]] 2023-02-13 16:53:57,641 INFO [Listener at localhost/45077] client.ConnectionUtils(127): regionserver/jenkins-hbase4:39009 server-side Connection retries=45 2023-02-13 16:53:57,641 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,642 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,642 INFO [Listener at localhost/45077] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 16:53:57,642 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 16:53:57,643 INFO [Listener at localhost/45077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 16:53:57,643 INFO [Listener at localhost/45077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 16:53:57,644 INFO [Listener at localhost/45077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39009 2023-02-13 16:53:57,645 INFO [Listener at localhost/45077] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 16:53:57,649 DEBUG [Listener at localhost/45077] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-13 16:53:57,649 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:57,650 INFO [Listener at localhost/45077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 16:53:57,651 INFO [Listener at localhost/45077] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39009 connecting to ZooKeeper ensemble=127.0.0.1:61694 2023-02-13 16:53:57,715 INFO [jenkins-hbase4:42765] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 16:53:57,717 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46119,1676307237115, state=OPENING 2023-02-13 16:53:57,719 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:390090x0, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 16:53:57,722 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 16:53:57,722 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(162): regionserver:390090x0, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 16:53:57,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39009-0x1004e9a5b35000f connected 2023-02-13 16:53:57,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46119,1676307237115}] 2023-02-13 16:53:57,722 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 16:53:57,724 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(162): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-02-13 16:53:57,728 INFO [RS:4;jenkins-hbase4:39253] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39253%2C1676307237520, suffix=, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307237520, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs, maxLogs=32 2023-02-13 16:53:57,732 DEBUG [Listener at localhost/45077] zookeeper.ZKUtil(164): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 16:53:57,732 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39009 2023-02-13 16:53:57,733 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39009 2023-02-13 16:53:57,733 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39009 2023-02-13 16:53:57,740 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39009 2023-02-13 16:53:57,744 DEBUG [Listener at localhost/45077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39009 2023-02-13 16:53:57,750 INFO [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(952): ClusterId : 1d915b4a-57b4-498c-b754-ef7b45ac1780 2023-02-13 16:53:57,750 DEBUG [RS:5;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 16:53:57,752 DEBUG [RS-EventLoopGroup-15-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:57,752 DEBUG [RS-EventLoopGroup-15-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:57,754 DEBUG [RS:5;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 16:53:57,754 DEBUG [RS:5;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 16:53:57,757 DEBUG [RS:5;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 16:53:57,760 DEBUG [RS-EventLoopGroup-15-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:57,765 DEBUG [RS:5;jenkins-hbase4:39009] zookeeper.ReadOnlyZKClient(139): Connect 0x7bd804f9 to 127.0.0.1:61694 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 16:53:57,777 INFO [RS:4;jenkins-hbase4:39253] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307237520/jenkins-hbase4.apache.org%2C39253%2C1676307237520.1676307237728 2023-02-13 16:53:57,780 DEBUG [RS:4;jenkins-hbase4:39253] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK], DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK], DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK]] 2023-02-13 16:53:57,780 DEBUG [RS:5;jenkins-hbase4:39009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5868cef2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 16:53:57,781 DEBUG [RS:5;jenkins-hbase4:39009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a695823, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:53:57,790 DEBUG [RS:5;jenkins-hbase4:39009] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:5;jenkins-hbase4:39009 2023-02-13 16:53:57,790 INFO [RS:5;jenkins-hbase4:39009] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 16:53:57,790 INFO [RS:5;jenkins-hbase4:39009] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 16:53:57,790 DEBUG [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 16:53:57,791 INFO [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42765,1676307237078 with isa=jenkins-hbase4.apache.org/172.31.14.131:39009, startcode=1676307237641 2023-02-13 16:53:57,791 DEBUG [RS:5;jenkins-hbase4:39009] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 16:53:57,793 INFO [RS-EventLoopGroup-13-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52013, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.11 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 16:53:57,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:57,794 DEBUG [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76 2023-02-13 16:53:57,794 DEBUG [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37465 2023-02-13 16:53:57,794 DEBUG [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 16:53:57,796 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:57,796 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:57,796 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:53:57,796 DEBUG [RS:5;jenkins-hbase4:39009] zookeeper.ZKUtil(162): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:57,796 WARN [RS:5;jenkins-hbase4:39009] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 16:53:57,796 INFO [RS:5;jenkins-hbase4:39009] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:57,797 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39009,1676307237641] 2023-02-13 16:53:57,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,797 DEBUG [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:57,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:57,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:57,803 DEBUG [RS:5;jenkins-hbase4:39009] zookeeper.ZKUtil(162): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:57,803 DEBUG [RS:5;jenkins-hbase4:39009] zookeeper.ZKUtil(162): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,803 DEBUG [RS:5;jenkins-hbase4:39009] zookeeper.ZKUtil(162): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:57,804 DEBUG [RS:5;jenkins-hbase4:39009] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 16:53:57,805 INFO [RS:5;jenkins-hbase4:39009] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 16:53:57,806 INFO [RS:5;jenkins-hbase4:39009] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 16:53:57,806 INFO [RS:5;jenkins-hbase4:39009] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 16:53:57,806 INFO [RS:5;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,807 INFO [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 16:53:57,808 INFO [RS:5;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,808 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:39009, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,808 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:39009, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,809 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:39009, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,809 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,809 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:39009, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,809 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:39009, corePoolSize=2, maxPoolSize=2 2023-02-13 16:53:57,809 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:39009, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,809 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:39009, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,809 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:39009, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,809 DEBUG [RS:5;jenkins-hbase4:39009] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:39009, corePoolSize=1, maxPoolSize=1 2023-02-13 16:53:57,810 INFO [RS:5;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,810 INFO [RS:5;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,810 INFO [RS:5;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,817 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=513ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 16:53:57,817 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPENING, ts=1676307237722, server=jenkins-hbase4.apache.org,46119,1676307237115}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 16:53:57,823 INFO [RS:5;jenkins-hbase4:39009] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 16:53:57,823 INFO [RS:5;jenkins-hbase4:39009] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39009,1676307237641-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:57,835 INFO [RS:5;jenkins-hbase4:39009] regionserver.Replication(203): jenkins-hbase4.apache.org,39009,1676307237641 started 2023-02-13 16:53:57,835 INFO [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39009,1676307237641, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39009, sessionid=0x1004e9a5b35000f 2023-02-13 16:53:57,835 DEBUG [RS:5;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 16:53:57,835 DEBUG [RS:5;jenkins-hbase4:39009] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:57,835 DEBUG [RS:5;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39009,1676307237641' 2023-02-13 16:53:57,835 DEBUG [RS:5;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 16:53:57,835 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:53:57,835 DEBUG [RS:5;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 16:53:57,836 DEBUG [RS:5;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 16:53:57,836 DEBUG [RS:5;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 16:53:57,836 DEBUG [RS:5;jenkins-hbase4:39009] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:57,836 DEBUG [RS:5;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39009,1676307237641' 2023-02-13 16:53:57,836 DEBUG [RS:5;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 16:53:57,836 DEBUG [RS:5;jenkins-hbase4:39009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 16:53:57,837 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55666, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:53:57,837 DEBUG [RS:5;jenkins-hbase4:39009] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 16:53:57,837 INFO [RS:5;jenkins-hbase4:39009] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 16:53:57,837 INFO [RS:5;jenkins-hbase4:39009] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 16:53:57,837 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46119] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Get size: 90 connection: 172.31.14.131:55666 deadline: 1676307297837, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,879 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:57,879 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:57,881 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55672, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-13 16:53:57,885 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 16:53:57,885 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 16:53:57,887 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46119%2C1676307237115.meta, suffix=.meta, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307237115, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs, maxLogs=32 2023-02-13 16:53:57,900 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:57,901 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:57,901 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:57,903 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307237115/jenkins-hbase4.apache.org%2C46119%2C1676307237115.meta.1676307237887.meta 2023-02-13 16:53:57,904 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK], DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK], DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK]] 2023-02-13 16:53:57,904 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:57,904 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 16:53:57,904 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 16:53:57,905 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 16:53:57,905 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 16:53:57,905 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:57,905 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 16:53:57,905 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 16:53:57,906 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 16:53:57,907 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info 2023-02-13 16:53:57,907 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info 2023-02-13 16:53:57,908 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 16:53:57,914 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info/f883d91573954b5aa8708e17bd64ea21 2023-02-13 16:53:57,914 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:57,914 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 16:53:57,915 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:57,915 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/rep_barrier 2023-02-13 16:53:57,916 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 16:53:57,916 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:57,917 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 16:53:57,917 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/table 2023-02-13 16:53:57,917 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/table 2023-02-13 16:53:57,918 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 16:53:57,924 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/table/dffd9fac4743453bb3301484d2d73476 2023-02-13 16:53:57,924 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:57,925 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740 2023-02-13 16:53:57,926 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740 2023-02-13 16:53:57,928 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 16:53:57,930 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 16:53:57,930 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=28; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62586402, jitterRate=-0.06738993525505066}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 16:53:57,930 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676307237905Writing region info on filesystem at 1676307237905Initializing all the Stores at 1676307237906 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307237906Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307237906Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307237906Cleaning up temporary data from old regions at 1676307237927 (+21 ms)Cleaning up detritus from prior splits at 1676307237928 (+1 ms)Running coprocessor post-open hooks at 1676307237930 (+2 ms)Region opened successfully at 1676307237930 2023-02-13 16:53:57,931 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=22, masterSystemTime=1676307237879 2023-02-13 16:53:57,936 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 16:53:57,936 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:46119-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 16:53:57,937 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46119,1676307237115, state=OPEN 2023-02-13 16:53:57,939 INFO [RS:5;jenkins-hbase4:39009] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39009%2C1676307237641, suffix=, logDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307237641, archiveDir=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs, maxLogs=32 2023-02-13 16:53:57,940 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 16:53:57,940 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 16:53:57,944 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-02-13 16:53:57,944 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46119,1676307237115 in 218 msec 2023-02-13 16:53:57,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-02-13 16:53:57,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 638 msec 2023-02-13 16:53:57,953 WARN [RPCClient-NioEventLoopGroup-6-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39061 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39061 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:57,953 WARN [Listener at localhost/45077] client.ConnectionImplementation(411): Checking master connection java.net.ConnectException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:39061 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39061 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:186) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:99) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:195) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$300(NettyRpcConnection.java:76) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:296) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:287) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39061 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 16:53:57,953 DEBUG [RPCClient-NioEventLoopGroup-6-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39061 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39061 2023-02-13 16:53:57,954 DEBUG [Listener at localhost/45077] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39061 this server is in the failed servers list 2023-02-13 16:53:57,955 WARN [Listener at localhost/45077] client.ConnectionImplementation(411): Checking master connection org.apache.hadoop.hbase.ipc.FailedServerException: Call to address=jenkins-hbase4.apache.org/172.31.14.131:39061 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:39061 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:236) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:417) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:413) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:430) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:326) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:582) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1174) at org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:404) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1945) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1383) at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1373) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:542) at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:228) at org.apache.hadoop.hbase.HBaseTestingUtility$9.evaluate(HBaseTestingUtility.java:3983) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134) at org.apache.hadoop.hbase.HBaseCommonTestingUtility.waitFor(HBaseCommonTestingUtility.java:234) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3136) at org.apache.hadoop.hbase.HBaseTestingUtility.waitTableAvailable(HBaseTestingUtility.java:3121) at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart.testRetainAssignmentOnClusterRestart(TestRetainAssignmentOnRestart.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: jenkins-hbase4.apache.org/172.31.14.131:39061 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:427) ... 46 more 2023-02-13 16:53:57,956 DEBUG [Listener at localhost/45077] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 16:53:57,958 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40296, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 16:53:57,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:57,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.CallRunner(144): callId: 49 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:40296 deadline: 1676307297959, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 16:53:57,963 DEBUG [RS-EventLoopGroup-16-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK] 2023-02-13 16:53:57,963 DEBUG [RS-EventLoopGroup-16-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK] 2023-02-13 16:53:57,963 DEBUG [RS-EventLoopGroup-16-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK] 2023-02-13 16:53:57,965 INFO [RS:5;jenkins-hbase4:39009] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307237641/jenkins-hbase4.apache.org%2C39009%2C1676307237641.1676307237939 2023-02-13 16:53:57,965 DEBUG [RS:5;jenkins-hbase4:39009] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35077,DS-9fbae8f7-cfec-455a-bf28-372062ff5846,DISK], DatanodeInfoWithStorage[127.0.0.1:39159,DS-dfc43236-f482-43c9-80b4-6c5055277247,DISK], DatanodeInfoWithStorage[127.0.0.1:43329,DS-dd1e8a26-d556-46eb-8cbd-2ec66d8cca9e,DISK]] 2023-02-13 16:53:58,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:58,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.CallRunner(144): callId: 51 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:40296 deadline: 1676307298060, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 16:53:58,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:58,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.CallRunner(144): callId: 53 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:40296 deadline: 1676307298263, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 16:53:58,270 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-13 16:53:58,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:58,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.CallRunner(144): callId: 55 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:40296 deadline: 1676307298566, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 16:53:58,817 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 16:53:58,820 DEBUG [hconnection-0x71bf9fc5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:53:58,822 INFO [RS-EventLoopGroup-14-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55678, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=ClientService 2023-02-13 16:53:58,825 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=745e4bc8b1326452bfdd5dbda9c99aeb, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,46119,1676307231186, regionLocation=jenkins-hbase4.apache.org,46119,1676307231186, openSeqNum=2 2023-02-13 16:53:58,825 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=c49f69a5b49e9a6b371b23ed03123cca, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39009,1676307231248, regionLocation=jenkins-hbase4.apache.org,39009,1676307231248, openSeqNum=2 2023-02-13 16:53:58,825 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=82053ffc9c655280d2c11277837c834b, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39253,1676307231215, regionLocation=jenkins-hbase4.apache.org,39253,1676307231215, openSeqNum=2 2023-02-13 16:53:58,825 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=b1fad04018b6f3dcf4291cca06397285, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39253,1676307231215, regionLocation=jenkins-hbase4.apache.org,39253,1676307231215, openSeqNum=2 2023-02-13 16:53:58,826 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 16:53:58,826 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676307298826 2023-02-13 16:53:58,826 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676307358826 2023-02-13 16:53:58,826 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 8 msec 2023-02-13 16:53:58,846 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,46119,1676307231186 had 2 regions 2023-02-13 16:53:58,846 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,39253,1676307231215 had 2 regions 2023-02-13 16:53:58,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42765,1676307237078-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:58,846 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,39009,1676307231248 had 1 regions 2023-02-13 16:53:58,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42765,1676307237078-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:58,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42765,1676307237078-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:58,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:42765, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:58,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 16:53:58,846 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1230): hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. is NOT online; state={745e4bc8b1326452bfdd5dbda9c99aeb state=OPEN, ts=1676307238825, server=jenkins-hbase4.apache.org,46119,1676307231186}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2023-02-13 16:53:58,847 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39009,1676307231248, splitWal=true, meta=false, isMeta: false 2023-02-13 16:53:58,848 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=20, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46119,1676307231186, splitWal=true, meta=true, isMeta: false 2023-02-13 16:53:58,847 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=18, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39253,1676307231215, splitWal=true, meta=false, isMeta: false 2023-02-13 16:53:58,849 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307231248-splitting 2023-02-13 16:53:58,850 WARN [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(172): unknown_server=jenkins-hbase4.apache.org,46119,1676307231186/hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb., unknown_server=jenkins-hbase4.apache.org,39009,1676307231248/restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca., unknown_server=jenkins-hbase4.apache.org,39253,1676307231215/restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b., unknown_server=jenkins-hbase4.apache.org,39253,1676307231215/restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:58,851 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307231248-splitting dir is empty, no logs to split. 2023-02-13 16:53:58,851 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39009,1676307231248 WAL count=0, meta=false 2023-02-13 16:53:58,852 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307231215-splitting 2023-02-13 16:53:58,852 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186-splitting dir is empty, no logs to split. 2023-02-13 16:53:58,852 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,46119,1676307231186 WAL count=0, meta=false 2023-02-13 16:53:58,853 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307231215-splitting dir is empty, no logs to split. 2023-02-13 16:53:58,853 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,39253,1676307231215 WAL count=0, meta=false 2023-02-13 16:53:58,853 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307231248-splitting dir is empty, no logs to split. 2023-02-13 16:53:58,853 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39009,1676307231248 WAL count=0, meta=false 2023-02-13 16:53:58,853 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39009,1676307231248 WAL splitting is done? wals=0, meta=false 2023-02-13 16:53:58,854 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,46119,1676307231186-splitting dir is empty, no logs to split. 2023-02-13 16:53:58,854 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,46119,1676307231186 WAL count=0, meta=false 2023-02-13 16:53:58,854 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,46119,1676307231186 WAL splitting is done? wals=0, meta=false 2023-02-13 16:53:58,855 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307231215-splitting dir is empty, no logs to split. 2023-02-13 16:53:58,855 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,39253,1676307231215 WAL count=0, meta=false 2023-02-13 16:53:58,855 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39253,1676307231215 WAL splitting is done? wals=0, meta=false 2023-02-13 16:53:58,855 INFO [PEWorker-2] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,39009,1676307231248 failed, ignore...File hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39009,1676307231248-splitting does not exist. 2023-02-13 16:53:58,856 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=745e4bc8b1326452bfdd5dbda9c99aeb, ASSIGN}] 2023-02-13 16:53:58,857 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=c49f69a5b49e9a6b371b23ed03123cca, ASSIGN}] 2023-02-13 16:53:58,857 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=745e4bc8b1326452bfdd5dbda9c99aeb, ASSIGN 2023-02-13 16:53:58,858 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableOne, region=c49f69a5b49e9a6b371b23ed03123cca, ASSIGN 2023-02-13 16:53:58,858 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46119,1676307231186 2023-02-13 16:53:58,858 INFO [PEWorker-4] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,39253,1676307231215 failed, ignore...File hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/WALs/jenkins-hbase4.apache.org,39253,1676307231215-splitting does not exist. 2023-02-13 16:53:58,858 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=23, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=745e4bc8b1326452bfdd5dbda9c99aeb, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,46119,1676307231186; forceNewPlan=false, retain=true 2023-02-13 16:53:58,858 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39009,1676307231248 2023-02-13 16:53:58,859 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableOne, region=c49f69a5b49e9a6b371b23ed03123cca, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,39009,1676307231248; forceNewPlan=false, retain=true 2023-02-13 16:53:58,859 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=b1fad04018b6f3dcf4291cca06397285, ASSIGN}, {pid=26, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=82053ffc9c655280d2c11277837c834b, ASSIGN}] 2023-02-13 16:53:58,860 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableTwo, region=b1fad04018b6f3dcf4291cca06397285, ASSIGN 2023-02-13 16:53:58,860 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=restartTableThree, region=82053ffc9c655280d2c11277837c834b, ASSIGN 2023-02-13 16:53:58,861 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:58,861 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableTwo, region=b1fad04018b6f3dcf4291cca06397285, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,39253,1676307231215; forceNewPlan=false, retain=true 2023-02-13 16:53:58,861 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39253,1676307231215 2023-02-13 16:53:58,861 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=restartTableThree, region=82053ffc9c655280d2c11277837c834b, ASSIGN; state=OPEN, location=jenkins-hbase4.apache.org,39253,1676307231215; forceNewPlan=false, retain=true 2023-02-13 16:53:59,008 INFO [jenkins-hbase4:42765] balancer.BaseLoadBalancer(1545): Reassigned 4 regions. 4 retained the pre-restart assignment. 2023-02-13 16:53:59,011 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=745e4bc8b1326452bfdd5dbda9c99aeb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:59,011 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=82053ffc9c655280d2c11277837c834b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:59,011 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b1fad04018b6f3dcf4291cca06397285, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:59,011 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=c49f69a5b49e9a6b371b23ed03123cca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:59,011 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307239011"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307239011"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307239011"}]},"ts":"1676307239011"} 2023-02-13 16:53:59,011 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307239011"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307239011"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307239011"}]},"ts":"1676307239011"} 2023-02-13 16:53:59,011 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307239011"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307239011"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307239011"}]},"ts":"1676307239011"} 2023-02-13 16:53:59,011 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307239011"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676307239011"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676307239011"}]},"ts":"1676307239011"} 2023-02-13 16:53:59,014 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 82053ffc9c655280d2c11277837c834b, server=jenkins-hbase4.apache.org,39253,1676307237520}] 2023-02-13 16:53:59,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=23, state=RUNNABLE; OpenRegionProcedure 745e4bc8b1326452bfdd5dbda9c99aeb, server=jenkins-hbase4.apache.org,46119,1676307237115}] 2023-02-13 16:53:59,016 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=24, state=RUNNABLE; OpenRegionProcedure c49f69a5b49e9a6b371b23ed03123cca, server=jenkins-hbase4.apache.org,39009,1676307237641}] 2023-02-13 16:53:59,017 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=25, state=RUNNABLE; OpenRegionProcedure b1fad04018b6f3dcf4291cca06397285, server=jenkins-hbase4.apache.org,39253,1676307237520}] 2023-02-13 16:53:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2823) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1073) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 16:53:59,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42765] ipc.CallRunner(144): callId: 57 service: MasterService methodName: GetTableDescriptors size: 61 connection: 172.31.14.131:40296 deadline: 1676307299071, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-02-13 16:53:59,166 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:59,166 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:59,167 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44274, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-13 16:53:59,169 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:59,169 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 16:53:59,171 INFO [RS-EventLoopGroup-16-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49956, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.8 (auth:SIMPLE), service=AdminService 2023-02-13 16:53:59,172 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:59,172 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(7850): Opening region: {ENCODED => 745e4bc8b1326452bfdd5dbda9c99aeb, NAME => 'hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:59,172 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:59,173 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:59,173 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(7890): checking encryption for 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:59,173 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(7893): checking classloading for 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:59,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] handler.AssignRegionHandler(128): Open restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:59,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] handler.AssignRegionHandler(128): Open restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:59,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(7850): Opening region: {ENCODED => b1fad04018b6f3dcf4291cca06397285, NAME => 'restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:59,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(7850): Opening region: {ENCODED => c49f69a5b49e9a6b371b23ed03123cca, NAME => 'restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:59,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableTwo b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:59,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableOne c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:59,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(865): Instantiated restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:59,180 INFO [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:59,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(7890): checking encryption for b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:59,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(7893): checking classloading for b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:59,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(865): Instantiated restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:59,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(7890): checking encryption for c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:59,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(7893): checking classloading for c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:59,181 DEBUG [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/info 2023-02-13 16:53:59,182 DEBUG [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/info 2023-02-13 16:53:59,182 INFO [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 745e4bc8b1326452bfdd5dbda9c99aeb columnFamilyName info 2023-02-13 16:53:59,184 INFO [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:59,185 INFO [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:59,185 DEBUG [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285/family 2023-02-13 16:53:59,186 DEBUG [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285/family 2023-02-13 16:53:59,186 DEBUG [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca/family 2023-02-13 16:53:59,186 DEBUG [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca/family 2023-02-13 16:53:59,186 INFO [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b1fad04018b6f3dcf4291cca06397285 columnFamilyName family 2023-02-13 16:53:59,186 INFO [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c49f69a5b49e9a6b371b23ed03123cca columnFamilyName family 2023-02-13 16:53:59,187 INFO [StoreOpener-b1fad04018b6f3dcf4291cca06397285-1] regionserver.HStore(310): Store=b1fad04018b6f3dcf4291cca06397285/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:59,187 INFO [StoreOpener-c49f69a5b49e9a6b371b23ed03123cca-1] regionserver.HStore(310): Store=c49f69a5b49e9a6b371b23ed03123cca/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:59,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:59,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:59,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:59,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:59,190 DEBUG [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] regionserver.HStore(539): loaded hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/info/794b2c13eb0644c099a6891718649f3f 2023-02-13 16:53:59,190 INFO [StoreOpener-745e4bc8b1326452bfdd5dbda9c99aeb-1] regionserver.HStore(310): Store=745e4bc8b1326452bfdd5dbda9c99aeb/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:59,191 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:59,192 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:59,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1054): writing seq id for b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:53:59,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(1054): writing seq id for c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:53:59,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(1071): Opened c49f69a5b49e9a6b371b23ed03123cca; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66434919, jitterRate=-0.010042563080787659}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:59,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1071): Opened b1fad04018b6f3dcf4291cca06397285; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58921620, jitterRate=-0.12199944257736206}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:59,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(964): Region open journal for c49f69a5b49e9a6b371b23ed03123cca: Running coprocessor pre-open hook at 1676307239181Writing region info on filesystem at 1676307239181Initializing all the Stores at 1676307239182 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307239182Cleaning up temporary data from old regions at 1676307239190 (+8 ms)Cleaning up detritus from prior splits at 1676307239190Running coprocessor post-open hooks at 1676307239194 (+4 ms)Region opened successfully at 1676307239194 2023-02-13 16:53:59,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(964): Region open journal for b1fad04018b6f3dcf4291cca06397285: Running coprocessor pre-open hook at 1676307239181Writing region info on filesystem at 1676307239181Initializing all the Stores at 1676307239181Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307239181Cleaning up temporary data from old regions at 1676307239190 (+9 ms)Cleaning up detritus from prior splits at 1676307239190Running coprocessor post-open hooks at 1676307239194 (+4 ms)Region opened successfully at 1676307239194 2023-02-13 16:53:59,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285., pid=30, masterSystemTime=1676307239166 2023-02-13 16:53:59,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca., pid=29, masterSystemTime=1676307239169 2023-02-13 16:53:59,197 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1054): writing seq id for 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:53:59,198 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1071): Opened 745e4bc8b1326452bfdd5dbda9c99aeb; next sequenceid=10; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69712849, jitterRate=0.03880240023136139}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:59,198 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(964): Region open journal for 745e4bc8b1326452bfdd5dbda9c99aeb: Running coprocessor pre-open hook at 1676307239173Writing region info on filesystem at 1676307239173Initializing all the Stores at 1676307239180 (+7 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676307239180Cleaning up temporary data from old regions at 1676307239192 (+12 ms)Cleaning up detritus from prior splits at 1676307239193 (+1 ms)Running coprocessor post-open hooks at 1676307239198 (+5 ms)Region opened successfully at 1676307239198 2023-02-13 16:53:59,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:59,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] handler.AssignRegionHandler(156): Opened restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:53:59,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] handler.AssignRegionHandler(128): Open restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:59,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(7850): Opening region: {ENCODED => 82053ffc9c655280d2c11277837c834b, NAME => 'restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.', STARTKEY => '', ENDKEY => ''} 2023-02-13 16:53:59,200 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb., pid=28, masterSystemTime=1676307239167 2023-02-13 16:53:59,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table restartTableThree 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:59,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(865): Instantiated restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 16:53:59,200 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b1fad04018b6f3dcf4291cca06397285, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:59,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(7890): checking encryption for 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:59,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(7893): checking classloading for 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:59,201 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307239200"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307239200"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307239200"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307239200"}]},"ts":"1676307239200"} 2023-02-13 16:53:59,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:59,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39009-0] handler.AssignRegionHandler(156): Opened restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:53:59,202 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=c49f69a5b49e9a6b371b23ed03123cca, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:53:59,202 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":"1676307239202"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307239202"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307239202"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307239202"}]},"ts":"1676307239202"} 2023-02-13 16:53:59,203 INFO [StoreOpener-82053ffc9c655280d2c11277837c834b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family family of region 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:59,203 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:59,203 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:46119-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:53:59,205 DEBUG [StoreOpener-82053ffc9c655280d2c11277837c834b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b/family 2023-02-13 16:53:59,205 DEBUG [StoreOpener-82053ffc9c655280d2c11277837c834b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b/family 2023-02-13 16:53:59,205 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=745e4bc8b1326452bfdd5dbda9c99aeb, regionState=OPEN, openSeqNum=10, regionLocation=jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:53:59,205 INFO [StoreOpener-82053ffc9c655280d2c11277837c834b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 82053ffc9c655280d2c11277837c834b columnFamilyName family 2023-02-13 16:53:59,205 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676307239205"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307239205"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307239205"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307239205"}]},"ts":"1676307239205"} 2023-02-13 16:53:59,206 INFO [StoreOpener-82053ffc9c655280d2c11277837c834b-1] regionserver.HStore(310): Store=82053ffc9c655280d2c11277837c834b/family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 16:53:59,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=25 2023-02-13 16:53:59,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=25, state=SUCCESS; OpenRegionProcedure b1fad04018b6f3dcf4291cca06397285, server=jenkins-hbase4.apache.org,39253,1676307237520 in 187 msec 2023-02-13 16:53:59,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:59,208 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=24 2023-02-13 16:53:59,208 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=24, state=SUCCESS; OpenRegionProcedure c49f69a5b49e9a6b371b23ed03123cca, server=jenkins-hbase4.apache.org,39009,1676307237641 in 189 msec 2023-02-13 16:53:59,209 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableTwo, region=b1fad04018b6f3dcf4291cca06397285, ASSIGN in 348 msec 2023-02-13 16:53:59,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:59,211 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=23 2023-02-13 16:53:59,211 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=23, state=SUCCESS; OpenRegionProcedure 745e4bc8b1326452bfdd5dbda9c99aeb, server=jenkins-hbase4.apache.org,46119,1676307237115 in 192 msec 2023-02-13 16:53:59,212 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=19 2023-02-13 16:53:59,212 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,39009,1676307231248 after splitting done 2023-02-13 16:53:59,212 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=restartTableOne, region=c49f69a5b49e9a6b371b23ed03123cca, ASSIGN in 351 msec 2023-02-13 16:53:59,212 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase4.apache.org,39009,1676307231248 from processing; numProcessing=2 2023-02-13 16:53:59,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1054): writing seq id for 82053ffc9c655280d2c11277837c834b 2023-02-13 16:53:59,215 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=20 2023-02-13 16:53:59,216 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=745e4bc8b1326452bfdd5dbda9c99aeb, ASSIGN in 355 msec 2023-02-13 16:53:59,216 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,46119,1676307231186 after splitting done 2023-02-13 16:53:59,216 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,46119,1676307231186 from processing; numProcessing=1 2023-02-13 16:53:59,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1071): Opened 82053ffc9c655280d2c11277837c834b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61167381, jitterRate=-0.08853499591350555}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 16:53:59,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(964): Region open journal for 82053ffc9c655280d2c11277837c834b: Running coprocessor pre-open hook at 1676307239201Writing region info on filesystem at 1676307239201Initializing all the Stores at 1676307239202 (+1 ms)Instantiating store for column family {NAME => 'family', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676307239203 (+1 ms)Cleaning up temporary data from old regions at 1676307239210 (+7 ms)Cleaning up detritus from prior splits at 1676307239211 (+1 ms)Running coprocessor post-open hooks at 1676307239216 (+5 ms)Region opened successfully at 1676307239216 2023-02-13 16:53:59,216 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,39009,1676307231248, splitWal=true, meta=false in 1.9660 sec 2023-02-13 16:53:59,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegionServer(2335): Post open deploy tasks for restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b., pid=27, masterSystemTime=1676307239166 2023-02-13 16:53:59,218 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,46119,1676307231186, splitWal=true, meta=true in 1.9660 sec 2023-02-13 16:53:59,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegionServer(2362): Finished post open deploy task for restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:59,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:39253-0] handler.AssignRegionHandler(156): Opened restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:53:59,219 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=82053ffc9c655280d2c11277837c834b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:53:59,220 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1676307239219"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676307239219"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676307239219"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676307239219"}]},"ts":"1676307239219"} 2023-02-13 16:53:59,224 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-02-13 16:53:59,224 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 82053ffc9c655280d2c11277837c834b, server=jenkins-hbase4.apache.org,39253,1676307237520 in 207 msec 2023-02-13 16:53:59,226 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=18 2023-02-13 16:53:59,226 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,39253,1676307231215 after splitting done 2023-02-13 16:53:59,226 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=restartTableThree, region=82053ffc9c655280d2c11277837c834b, ASSIGN in 365 msec 2023-02-13 16:53:59,226 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,39253,1676307231215 from processing; numProcessing=0 2023-02-13 16:53:59,228 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,39253,1676307231215, splitWal=true, meta=false in 1.9820 sec 2023-02-13 16:53:59,847 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-02-13 16:53:59,862 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 16:53:59,865 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 16:53:59,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.757sec 2023-02-13 16:53:59,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 16:53:59,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 16:53:59,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 16:53:59,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42765,1676307237078-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 16:53:59,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42765,1676307237078-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 16:53:59,868 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 16:54:00,080 DEBUG [hconnection-0x77a00002-shared-pool-6] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:54:00,081 INFO [RS-EventLoopGroup-16-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49972, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:54:00,082 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:54:00,090 DEBUG [hconnection-0x77a00002-shared-pool-9] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 16:54:00,091 INFO [RS-EventLoopGroup-15-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44276, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 16:54:00,092 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:54:00,099 INFO [Listener at localhost/45077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 16:54:00,099 INFO [Listener at localhost/45077] master.SnapshotOfRegionAssignmentFromMeta(102): Start to scan the hbase:meta for the current region assignment snappshot 2023-02-13 16:54:00,102 INFO [Listener at localhost/45077] master.SnapshotOfRegionAssignmentFromMeta(164): Finished to scan the hbase:meta for the current region assignmentsnapshot 2023-02-13 16:54:00,102 INFO [Listener at localhost/45077] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => 745e4bc8b1326452bfdd5dbda9c99aeb, NAME => 'hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,46119,1676307231186, currentServer=jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:54:00,102 INFO [Listener at localhost/45077] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => c49f69a5b49e9a6b371b23ed03123cca, NAME => 'restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,39009,1676307231248, currentServer=jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:54:00,102 INFO [Listener at localhost/45077] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => b1fad04018b6f3dcf4291cca06397285, NAME => 'restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,39253,1676307231215, currentServer=jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:54:00,102 INFO [Listener at localhost/45077] master.TestRetainAssignmentOnRestart(119): Key={ENCODED => 82053ffc9c655280d2c11277837c834b, NAME => 'restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.', STARTKEY => '', ENDKEY => ''} oldServer=jenkins-hbase4.apache.org,39253,1676307231215, currentServer=jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:54:00,102 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 16:54:00,102 INFO [Listener at localhost/45077] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 16:54:00,102 DEBUG [Listener at localhost/45077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58661f63 to 127.0.0.1:61694 2023-02-13 16:54:00,103 DEBUG [Listener at localhost/45077] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:54:00,103 DEBUG [Listener at localhost/45077] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 16:54:00,103 DEBUG [Listener at localhost/45077] util.JVMClusterUtil(257): Found active master hash=1942751537, stopped=false 2023-02-13 16:54:00,103 INFO [Listener at localhost/45077] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:54:00,106 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 16:54:00,106 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 16:54:00,106 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 16:54:00,106 INFO [Listener at localhost/45077] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 16:54:00,106 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 16:54:00,106 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:54:00,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:54:00,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:54:00,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:54:00,109 DEBUG [Listener at localhost/45077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37ca281d to 127.0.0.1:61694 2023-02-13 16:54:00,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 16:54:00,109 DEBUG [Listener at localhost/45077] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:54:00,109 INFO [Listener at localhost/45077] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46119,1676307237115' ***** 2023-02-13 16:54:00,109 INFO [Listener at localhost/45077] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 16:54:00,109 INFO [Listener at localhost/45077] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39253,1676307237520' ***** 2023-02-13 16:54:00,109 INFO [Listener at localhost/45077] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 16:54:00,109 INFO [Listener at localhost/45077] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39009,1676307237641' ***** 2023-02-13 16:54:00,109 INFO [Listener at localhost/45077] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 16:54:00,109 INFO [RS:4;jenkins-hbase4:39253] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:54:00,110 INFO [RS:4;jenkins-hbase4:39253] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:54:00,110 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:54:00,110 INFO [RS:4;jenkins-hbase4:39253] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:54:00,110 INFO [RS:3;jenkins-hbase4:46119] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:54:00,111 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:54:00,110 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(3304): Received CLOSE for b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:54:00,111 INFO [RS:5;jenkins-hbase4:39009] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 16:54:00,111 INFO [RS:3;jenkins-hbase4:46119] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:54:00,111 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(3304): Received CLOSE for 82053ffc9c655280d2c11277837c834b 2023-02-13 16:54:00,111 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 16:54:00,112 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:54:00,111 INFO [RS:3;jenkins-hbase4:46119] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:54:00,111 INFO [RS:5;jenkins-hbase4:39009] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 16:54:00,113 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(3304): Received CLOSE for 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:54:00,113 INFO [RS:5;jenkins-hbase4:39009] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 16:54:00,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1603): Closing b1fad04018b6f3dcf4291cca06397285, disabling compactions & flushes 2023-02-13 16:54:00,112 DEBUG [RS:4;jenkins-hbase4:39253] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3defb754 to 127.0.0.1:61694 2023-02-13 16:54:00,113 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:54:00,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1625): Closing region restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:54:00,113 DEBUG [RS:3;jenkins-hbase4:46119] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d8d4df1 to 127.0.0.1:61694 2023-02-13 16:54:00,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1603): Closing 745e4bc8b1326452bfdd5dbda9c99aeb, disabling compactions & flushes 2023-02-13 16:54:00,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:54:00,113 INFO [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(3304): Received CLOSE for c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:54:00,114 INFO [regionserver/jenkins-hbase4:39009.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:54:00,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:54:00,113 DEBUG [RS:3;jenkins-hbase4:46119] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:54:00,114 INFO [regionserver/jenkins-hbase4:39253.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:54:00,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:54:00,113 DEBUG [RS:4;jenkins-hbase4:39253] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:54:00,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1713): Acquired close lock on restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. after waiting 0 ms 2023-02-13 16:54:00,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(1603): Closing c49f69a5b49e9a6b371b23ed03123cca, disabling compactions & flushes 2023-02-13 16:54:00,114 INFO [RS:3;jenkins-hbase4:46119] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:54:00,114 INFO [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:54:00,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. after waiting 0 ms 2023-02-13 16:54:00,114 INFO [regionserver/jenkins-hbase4:46119.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:54:00,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:54:00,115 DEBUG [RS:5;jenkins-hbase4:39009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bd804f9 to 127.0.0.1:61694 2023-02-13 16:54:00,115 INFO [RS:3;jenkins-hbase4:46119] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:54:00,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(1625): Closing region restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:54:00,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:54:00,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(1713): Acquired close lock on restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. after waiting 0 ms 2023-02-13 16:54:00,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1723): Updates disabled for region restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:54:00,115 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 16:54:00,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(1723): Updates disabled for region restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:54:00,116 DEBUG [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1479): Online Regions={b1fad04018b6f3dcf4291cca06397285=restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285., 82053ffc9c655280d2c11277837c834b=restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b.} 2023-02-13 16:54:00,116 INFO [RS:3;jenkins-hbase4:46119] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:54:00,117 DEBUG [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1505): Waiting on 82053ffc9c655280d2c11277837c834b, b1fad04018b6f3dcf4291cca06397285 2023-02-13 16:54:00,117 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 16:54:00,116 DEBUG [RS:5;jenkins-hbase4:39009] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:54:00,117 INFO [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 16:54:00,118 DEBUG [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1479): Online Regions={c49f69a5b49e9a6b371b23ed03123cca=restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca.} 2023-02-13 16:54:00,118 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-13 16:54:00,118 DEBUG [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1505): Waiting on c49f69a5b49e9a6b371b23ed03123cca 2023-02-13 16:54:00,118 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 745e4bc8b1326452bfdd5dbda9c99aeb=hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb.} 2023-02-13 16:54:00,118 DEBUG [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1505): Waiting on 1588230740, 745e4bc8b1326452bfdd5dbda9c99aeb 2023-02-13 16:54:00,119 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 16:54:00,119 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 16:54:00,119 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 16:54:00,119 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 16:54:00,119 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 16:54:00,119 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=3.81 KB heapSize=6.91 KB 2023-02-13 16:54:00,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableOne/c49f69a5b49e9a6b371b23ed03123cca/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 16:54:00,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableTwo/b1fad04018b6f3dcf4291cca06397285/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 16:54:00,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/namespace/745e4bc8b1326452bfdd5dbda9c99aeb/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=9 2023-02-13 16:54:00,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(1837): Closed restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:54:00,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009-0] regionserver.HRegion(1557): Region close journal for c49f69a5b49e9a6b371b23ed03123cca: Waiting for close lock at 1676307240114Running coprocessor pre-close hooks at 1676307240114Disabling compacts and flushes for region at 1676307240114Disabling writes for close at 1676307240116 (+2 ms)Writing region close event to WAL at 1676307240118 (+2 ms)Running coprocessor post-close hooks at 1676307240123 (+5 ms)Closed at 1676307240123 2023-02-13 16:54:00,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1837): Closed restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:54:00,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39009-0] handler.CloseRegionHandler(117): Closed restartTableOne,,1676307232395.c49f69a5b49e9a6b371b23ed03123cca. 2023-02-13 16:54:00,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1557): Region close journal for b1fad04018b6f3dcf4291cca06397285: Waiting for close lock at 1676307240112Running coprocessor pre-close hooks at 1676307240112Disabling compacts and flushes for region at 1676307240112Disabling writes for close at 1676307240115 (+3 ms)Writing region close event to WAL at 1676307240118 (+3 ms)Running coprocessor post-close hooks at 1676307240123 (+5 ms)Closed at 1676307240123 2023-02-13 16:54:00,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] handler.CloseRegionHandler(117): Closed restartTableTwo,,1676307233010.b1fad04018b6f3dcf4291cca06397285. 2023-02-13 16:54:00,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1603): Closing 82053ffc9c655280d2c11277837c834b, disabling compactions & flushes 2023-02-13 16:54:00,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:54:00,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1625): Closing region restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:54:00,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1557): Region close journal for 745e4bc8b1326452bfdd5dbda9c99aeb: Waiting for close lock at 1676307240113Running coprocessor pre-close hooks at 1676307240113Disabling compacts and flushes for region at 1676307240113Disabling writes for close at 1676307240115 (+2 ms)Writing region close event to WAL at 1676307240118 (+3 ms)Running coprocessor post-close hooks at 1676307240124 (+6 ms)Closed at 1676307240125 (+1 ms) 2023-02-13 16:54:00,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1646): Waiting without time limit for close lock on restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:54:00,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:46119-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676307231874.745e4bc8b1326452bfdd5dbda9c99aeb. 2023-02-13 16:54:00,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1713): Acquired close lock on restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. after waiting 0 ms 2023-02-13 16:54:00,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1723): Updates disabled for region restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:54:00,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/default/restartTableThree/82053ffc9c655280d2c11277837c834b/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-13 16:54:00,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1837): Closed restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:54:00,130 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] regionserver.HRegion(1557): Region close journal for 82053ffc9c655280d2c11277837c834b: Waiting for close lock at 1676307240124Running coprocessor pre-close hooks at 1676307240124Disabling compacts and flushes for region at 1676307240124Disabling writes for close at 1676307240126 (+2 ms)Writing region close event to WAL at 1676307240126Running coprocessor post-close hooks at 1676307240130 (+4 ms)Closed at 1676307240130 2023-02-13 16:54:00,130 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:39253-0] handler.CloseRegionHandler(117): Closed restartTableThree,,1676307233625.82053ffc9c655280d2c11277837c834b. 2023-02-13 16:54:00,137 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.81 KB at sequenceid=38 (bloomFilter=false), to=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/.tmp/info/a39e772d581d4588825c044cd0a3a4cf 2023-02-13 16:54:00,143 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/.tmp/info/a39e772d581d4588825c044cd0a3a4cf as hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info/a39e772d581d4588825c044cd0a3a4cf 2023-02-13 16:54:00,149 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HStore(1080): Added hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/info/a39e772d581d4588825c044cd0a3a4cf, entries=32, sequenceid=38, filesize=8.5 K 2023-02-13 16:54:00,150 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(2947): Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 31ms, sequenceid=38, compaction requested=false 2023-02-13 16:54:00,158 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/data/hbase/meta/1588230740/recovered.edits/41.seqid, newMaxSeqId=41, maxSeqId=27 2023-02-13 16:54:00,158 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 16:54:00,159 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 16:54:00,159 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676307240119Running coprocessor pre-close hooks at 1676307240119Disabling compacts and flushes for region at 1676307240119Disabling writes for close at 1676307240119Obtaining lock to block concurrent updates at 1676307240119Preparing flush snapshotting stores in 1588230740 at 1676307240119Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=3900, getHeapSize=7032, getOffHeapSize=0, getCellsCount=32 at 1676307240120 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676307240120Flushing 1588230740/info: creating writer at 1676307240121 (+1 ms)Flushing 1588230740/info: appending metadata at 1676307240123 (+2 ms)Flushing 1588230740/info: closing flushed file at 1676307240123Flushing 1588230740/info: reopening flushed file at 1676307240144 (+21 ms)Finished flush of dataSize ~3.81 KB/3900, heapSize ~6.40 KB/6552, currentSize=0 B/0 for 1588230740 in 31ms, sequenceid=38, compaction requested=false at 1676307240150 (+6 ms)Writing region close event to WAL at 1676307240154 (+4 ms)Running coprocessor post-close hooks at 1676307240158 (+4 ms)Closed at 1676307240159 (+1 ms) 2023-02-13 16:54:00,159 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:46119-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 16:54:00,317 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39253,1676307237520; all regions closed. 2023-02-13 16:54:00,318 INFO [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39009,1676307237641; all regions closed. 2023-02-13 16:54:00,318 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46119,1676307237115; all regions closed. 2023-02-13 16:54:00,324 DEBUG [RS:5;jenkins-hbase4:39009] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs 2023-02-13 16:54:00,324 INFO [RS:5;jenkins-hbase4:39009] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39009%2C1676307237641:(num 1676307237939) 2023-02-13 16:54:00,325 DEBUG [RS:5;jenkins-hbase4:39009] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:54:00,325 INFO [RS:5;jenkins-hbase4:39009] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:54:00,325 INFO [RS:5;jenkins-hbase4:39009] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:39009 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 16:54:00,325 INFO [RS:5;jenkins-hbase4:39009] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:54:00,326 INFO [regionserver/jenkins-hbase4:39009.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:54:00,326 INFO [RS:5;jenkins-hbase4:39009] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:54:00,326 INFO [RS:5;jenkins-hbase4:39009] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:54:00,327 INFO [RS:5;jenkins-hbase4:39009] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39009 2023-02-13 16:54:00,328 DEBUG [RS:4;jenkins-hbase4:39253] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs 2023-02-13 16:54:00,329 INFO [RS:4;jenkins-hbase4:39253] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39253%2C1676307237520:(num 1676307237728) 2023-02-13 16:54:00,329 DEBUG [RS:4;jenkins-hbase4:39253] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:54:00,329 INFO [RS:4;jenkins-hbase4:39253] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:54:00,329 INFO [RS:4;jenkins-hbase4:39253] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:39253 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 16:54:00,330 INFO [RS:4;jenkins-hbase4:39253] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 16:54:00,330 INFO [regionserver/jenkins-hbase4:39253.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:54:00,330 INFO [RS:4;jenkins-hbase4:39253] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 16:54:00,330 INFO [RS:4;jenkins-hbase4:39253] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 16:54:00,330 INFO [RS:4;jenkins-hbase4:39253] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39253 2023-02-13 16:54:00,331 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:54:00,331 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:54:00,331 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:54:00,332 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@29302ddd rejected from java.util.concurrent.ThreadPoolExecutor@2d0fac23[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,332 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39009,1676307237641 2023-02-13 16:54:00,332 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:54:00,331 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:54:00,332 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@71d9b8eb rejected from java.util.concurrent.ThreadPoolExecutor@2d0fac23[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,332 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:54:00,333 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:54:00,333 DEBUG [RS:3;jenkins-hbase4:46119] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs 2023-02-13 16:54:00,333 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:54:00,333 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39253,1676307237520 2023-02-13 16:54:00,333 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@911ec7a rejected from java.util.concurrent.ThreadPoolExecutor@2d0fac23[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,333 INFO [RS:3;jenkins-hbase4:46119] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46119%2C1676307237115.meta:.meta(num 1676307237887) 2023-02-13 16:54:00,333 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20be0f0 rejected from java.util.concurrent.ThreadPoolExecutor@6347469[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,334 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39253,1676307237520] 2023-02-13 16:54:00,334 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39253,1676307237520; numProcessing=1 2023-02-13 16:54:00,338 DEBUG [RS:3;jenkins-hbase4:46119] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/oldWALs 2023-02-13 16:54:00,338 INFO [RS:3;jenkins-hbase4:46119] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46119%2C1676307237115:(num 1676307237613) 2023-02-13 16:54:00,338 DEBUG [RS:3;jenkins-hbase4:46119] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:54:00,338 INFO [RS:3;jenkins-hbase4:46119] regionserver.LeaseManager(133): Closed leases 2023-02-13 16:54:00,338 INFO [RS:3;jenkins-hbase4:46119] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:46119 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 16:54:00,338 INFO [regionserver/jenkins-hbase4:46119.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:54:00,339 INFO [RS:3;jenkins-hbase4:46119] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46119 2023-02-13 16:54:00,434 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:54:00,434 INFO [RS:5;jenkins-hbase4:39009] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39009,1676307237641; zookeeper connection closed. 2023-02-13 16:54:00,434 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@348037a6 rejected from java.util.concurrent.ThreadPoolExecutor@2d0fac23[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,434 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39009-0x1004e9a5b35000f, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:54:00,434 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@24954d03] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@24954d03 2023-02-13 16:54:00,434 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@12154159 rejected from java.util.concurrent.ThreadPoolExecutor@2d0fac23[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,436 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39253,1676307237520 already deleted, retry=false 2023-02-13 16:54:00,436 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46119,1676307237115 2023-02-13 16:54:00,436 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39253,1676307237520 expired; onlineServers=2 2023-02-13 16:54:00,436 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 16:54:00,436 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4d171251 rejected from java.util.concurrent.ThreadPoolExecutor@49289f2b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,436 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39009,1676307237641] 2023-02-13 16:54:00,437 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39009,1676307237641; numProcessing=2 2023-02-13 16:54:00,438 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39009,1676307237641 already deleted, retry=false 2023-02-13 16:54:00,438 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39009,1676307237641 expired; onlineServers=1 2023-02-13 16:54:00,438 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46119,1676307237115] 2023-02-13 16:54:00,439 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46119,1676307237115; numProcessing=3 2023-02-13 16:54:00,441 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46119,1676307237115 already deleted, retry=false 2023-02-13 16:54:00,441 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,46119,1676307237115 expired; onlineServers=0 2023-02-13 16:54:00,441 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42765,1676307237078' ***** 2023-02-13 16:54:00,441 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 16:54:00,442 DEBUG [M:1;jenkins-hbase4:42765] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77f38390, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 16:54:00,442 INFO [M:1;jenkins-hbase4:42765] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42765,1676307237078 2023-02-13 16:54:00,442 INFO [M:1;jenkins-hbase4:42765] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42765,1676307237078; all regions closed. 2023-02-13 16:54:00,442 DEBUG [M:1;jenkins-hbase4:42765] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 16:54:00,442 DEBUG [M:1;jenkins-hbase4:42765] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 16:54:00,442 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 16:54:00,442 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307237297] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676307237297,5,FailOnTimeoutGroup] 2023-02-13 16:54:00,442 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307237303] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676307237303,5,FailOnTimeoutGroup] 2023-02-13 16:54:00,442 DEBUG [M:1;jenkins-hbase4:42765] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 16:54:00,444 INFO [M:1;jenkins-hbase4:42765] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 16:54:00,444 INFO [M:1;jenkins-hbase4:42765] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 16:54:00,444 INFO [M:1;jenkins-hbase4:42765] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 16:54:00,444 DEBUG [M:1;jenkins-hbase4:42765] master.HMaster(1502): Stopping service threads 2023-02-13 16:54:00,444 INFO [M:1;jenkins-hbase4:42765] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 16:54:00,444 ERROR [M:1;jenkins-hbase4:42765] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-13 16:54:00,445 INFO [M:1;jenkins-hbase4:42765] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 16:54:00,445 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 16:54:00,445 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 16:54:00,445 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 16:54:00,445 DEBUG [M:1;jenkins-hbase4:42765] zookeeper.ZKUtil(398): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 16:54:00,445 WARN [M:1;jenkins-hbase4:42765] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 16:54:00,445 INFO [M:1;jenkins-hbase4:42765] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 16:54:00,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 16:54:00,445 INFO [M:1;jenkins-hbase4:42765] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 16:54:00,446 DEBUG [M:1;jenkins-hbase4:42765] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 16:54:00,446 INFO [M:1;jenkins-hbase4:42765] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:54:00,446 DEBUG [M:1;jenkins-hbase4:42765] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:54:00,446 DEBUG [M:1;jenkins-hbase4:42765] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 16:54:00,446 DEBUG [M:1;jenkins-hbase4:42765] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:54:00,446 INFO [M:1;jenkins-hbase4:42765] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=39.05 KB heapSize=48.34 KB 2023-02-13 16:54:00,458 INFO [M:1;jenkins-hbase4:42765] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.05 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/542ad1acb9cc40a4b199ecffa2bfa0ee 2023-02-13 16:54:00,463 DEBUG [M:1;jenkins-hbase4:42765] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/542ad1acb9cc40a4b199ecffa2bfa0ee as hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/542ad1acb9cc40a4b199ecffa2bfa0ee 2023-02-13 16:54:00,468 INFO [M:1;jenkins-hbase4:42765] regionserver.HStore(1080): Added hdfs://localhost:37465/user/jenkins/test-data/d8451c17-b398-d6a8-d843-2b13cceecc76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/542ad1acb9cc40a4b199ecffa2bfa0ee, entries=13, sequenceid=247, filesize=6.2 K 2023-02-13 16:54:00,469 INFO [M:1;jenkins-hbase4:42765] regionserver.HRegion(2947): Finished flush of dataSize ~39.05 KB/39989, heapSize ~48.32 KB/49480, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 23ms, sequenceid=247, compaction requested=false 2023-02-13 16:54:00,471 INFO [M:1;jenkins-hbase4:42765] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 16:54:00,471 DEBUG [M:1;jenkins-hbase4:42765] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676307240446Disabling compacts and flushes for region at 1676307240446Disabling writes for close at 1676307240446Obtaining lock to block concurrent updates at 1676307240446Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676307240446Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=39989, getHeapSize=49480, getOffHeapSize=0, getCellsCount=122 at 1676307240446Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676307240447 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676307240447Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676307240450 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676307240450Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676307240464 (+14 ms)Finished flush of dataSize ~39.05 KB/39989, heapSize ~48.32 KB/49480, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 23ms, sequenceid=247, compaction requested=false at 1676307240469 (+5 ms)Writing region close event to WAL at 1676307240471 (+2 ms)Closed at 1676307240471 2023-02-13 16:54:00,474 INFO [M:1;jenkins-hbase4:42765] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 16:54:00,474 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 16:54:00,475 INFO [M:1;jenkins-hbase4:42765] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42765 2023-02-13 16:54:00,477 DEBUG [M:1;jenkins-hbase4:42765] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,42765,1676307237078 already deleted, retry=false 2023-02-13 16:54:00,708 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:54:00,708 INFO [M:1;jenkins-hbase4:42765] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42765,1676307237078; zookeeper connection closed. 2023-02-13 16:54:00,708 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f3fa79c rejected from java.util.concurrent.ThreadPoolExecutor@4b003aa3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 19] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,709 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): master:42765-0x1004e9a5b350009, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:54:00,709 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@df7037d rejected from java.util.concurrent.ThreadPoolExecutor@4b003aa3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 19] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,808 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:54:00,808 INFO [RS:3;jenkins-hbase4:46119] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46119,1676307237115; zookeeper connection closed. 2023-02-13 16:54:00,809 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@481c4798 rejected from java.util.concurrent.ThreadPoolExecutor@49289f2b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,810 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@25fdc1a4] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@25fdc1a4 2023-02-13 16:54:00,810 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:46119-0x1004e9a5b35000a, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:54:00,811 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@30b05011 rejected from java.util.concurrent.ThreadPoolExecutor@49289f2b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,909 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:54:00,909 INFO [RS:4;jenkins-hbase4:39253] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39253,1676307237520; zookeeper connection closed. 2023-02-13 16:54:00,909 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2f6e1163 rejected from java.util.concurrent.ThreadPoolExecutor@6347469[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,909 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@564915c9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@564915c9 2023-02-13 16:54:00,909 DEBUG [Listener at localhost/45077-EventThread] zookeeper.ZKWatcher(600): regionserver:39253-0x1004e9a5b35000d, quorum=127.0.0.1:61694, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 16:54:00,909 ERROR [Listener at localhost/45077-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@570dcd1c rejected from java.util.concurrent.ThreadPoolExecutor@6347469[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 16:54:00,909 INFO [Listener at localhost/45077] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 6 regionserver(s) complete 2023-02-13 16:54:00,910 WARN [Listener at localhost/45077] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 16:54:00,915 INFO [Listener at localhost/45077] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 16:54:01,021 WARN [BP-1201217589-172.31.14.131-1676307230376 heartbeating to localhost/127.0.0.1:37465] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 16:54:01,021 WARN [BP-1201217589-172.31.14.131-1676307230376 heartbeating to localhost/127.0.0.1:37465] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1201217589-172.31.14.131-1676307230376 (Datanode Uuid e45dd6e8-bd82-41c6-a1be-6ba70c0c0a15) service to localhost/127.0.0.1:37465 2023-02-13 16:54:01,022 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945/dfs/data/data5/current/BP-1201217589-172.31.14.131-1676307230376] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:54:01,022 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945/dfs/data/data6/current/BP-1201217589-172.31.14.131-1676307230376] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:54:01,023 WARN [Listener at localhost/45077] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 16:54:01,027 INFO [Listener at localhost/45077] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 16:54:01,131 WARN [BP-1201217589-172.31.14.131-1676307230376 heartbeating to localhost/127.0.0.1:37465] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 16:54:01,131 WARN [BP-1201217589-172.31.14.131-1676307230376 heartbeating to localhost/127.0.0.1:37465] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1201217589-172.31.14.131-1676307230376 (Datanode Uuid 78d69e1e-33b1-458d-a2ad-bd3e66e638b0) service to localhost/127.0.0.1:37465 2023-02-13 16:54:01,132 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945/dfs/data/data3/current/BP-1201217589-172.31.14.131-1676307230376] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:54:01,132 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945/dfs/data/data4/current/BP-1201217589-172.31.14.131-1676307230376] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:54:01,133 WARN [Listener at localhost/45077] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 16:54:01,136 INFO [Listener at localhost/45077] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 16:54:01,241 WARN [BP-1201217589-172.31.14.131-1676307230376 heartbeating to localhost/127.0.0.1:37465] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 16:54:01,241 WARN [BP-1201217589-172.31.14.131-1676307230376 heartbeating to localhost/127.0.0.1:37465] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1201217589-172.31.14.131-1676307230376 (Datanode Uuid 47024625-c8f3-4dac-b947-85fd356dcf27) service to localhost/127.0.0.1:37465 2023-02-13 16:54:01,241 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945/dfs/data/data1/current/BP-1201217589-172.31.14.131-1676307230376] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:54:01,242 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ed809da8-c99b-94d6-8a76-e1fc9d989fbb/cluster_b0928926-cf29-1b6b-b1d0-b5a397fff945/dfs/data/data2/current/BP-1201217589-172.31.14.131-1676307230376] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 16:54:01,251 INFO [Listener at localhost/45077] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 16:54:01,367 INFO [Listener at localhost/45077] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 16:54:01,397 INFO [Listener at localhost/45077] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-13 16:54:01,414 INFO [Listener at localhost/45077] hbase.ResourceChecker(175): after: master.TestRetainAssignmentOnRestartSplitWithoutZk#testRetainAssignmentOnClusterRestart Thread=159 (was 102) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:37465 from jenkins.hfs.11 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:37465 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:37465 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/45077 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:37465 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:37465 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.9@localhost:37465 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:37465 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.7@localhost:37465 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.6@localhost:37465 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.11@localhost:37465 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:37465 from jenkins.hfs.8 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: regionserver/jenkins-hbase4:39253.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:37465 from jenkins.hfs.6 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:37465 from jenkins.hfs.9 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-15-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:37465 from jenkins.hfs.10 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.8@localhost:37465 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.10@localhost:37465 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:46119.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (821811218) connection to localhost/127.0.0.1:37465 from jenkins.hfs.7 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-13-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:39009.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) - Thread LEAK? -, OpenFileDescriptor=605 (was 516) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=387 (was 379) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=1354 (was 1591)