2023-07-18 05:05:14,506 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-18 05:05:14,735 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767 2023-07-18 05:05:14,747 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-18 05:05:14,748 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80, deleteOnExit=true 2023-07-18 05:05:14,748 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-18 05:05:14,748 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/test.cache.data in system properties and HBase conf 2023-07-18 05:05:14,749 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/hadoop.tmp.dir in system properties and HBase conf 2023-07-18 05:05:14,749 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/hadoop.log.dir in system properties and HBase conf 2023-07-18 05:05:14,750 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-18 05:05:14,750 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-18 05:05:14,750 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-18 05:05:14,866 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-18 05:05:15,251 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-18 05:05:15,255 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-18 05:05:15,255 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-18 05:05:15,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-18 05:05:15,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-18 05:05:15,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-18 05:05:15,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-18 05:05:15,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-18 05:05:15,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-18 05:05:15,258 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-18 05:05:15,258 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/nfs.dump.dir in system properties and HBase conf 2023-07-18 05:05:15,259 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/java.io.tmpdir in system properties and HBase conf 2023-07-18 05:05:15,259 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-18 05:05:15,259 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-18 05:05:15,259 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-18 05:05:15,737 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-18 05:05:15,741 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-18 05:05:16,006 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-18 05:05:16,159 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-18 05:05:16,174 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-18 05:05:16,208 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-18 05:05:16,240 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/java.io.tmpdir/Jetty_localhost_37879_hdfs____w1j3z4/webapp 2023-07-18 05:05:16,391 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37879 2023-07-18 05:05:16,403 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-18 05:05:16,403 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-18 05:05:16,772 WARN [Listener at localhost/36887] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-18 05:05:16,872 WARN [Listener at localhost/36887] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-18 05:05:16,890 WARN [Listener at localhost/36887] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-18 05:05:16,897 INFO [Listener at localhost/36887] log.Slf4jLog(67): jetty-6.1.26 2023-07-18 05:05:16,901 INFO [Listener at localhost/36887] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/java.io.tmpdir/Jetty_localhost_36797_datanode____5eoshl/webapp 2023-07-18 05:05:17,004 INFO [Listener at localhost/36887] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36797 2023-07-18 05:05:17,345 WARN [Listener at localhost/46529] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-18 05:05:17,356 WARN [Listener at localhost/46529] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-18 05:05:17,359 WARN [Listener at localhost/46529] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-18 05:05:17,361 INFO [Listener at localhost/46529] log.Slf4jLog(67): jetty-6.1.26 2023-07-18 05:05:17,366 INFO [Listener at localhost/46529] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/java.io.tmpdir/Jetty_localhost_45503_datanode____9zupfi/webapp 2023-07-18 05:05:17,471 INFO [Listener at localhost/46529] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45503 2023-07-18 05:05:17,484 WARN [Listener at localhost/35655] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-18 05:05:17,867 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3bcf145430bfd1a7: Processing first storage report for DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333 from datanode 3b0f0fec-8b12-4bb0-9296-5c650e0a5410 2023-07-18 05:05:17,868 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3bcf145430bfd1a7: from storage DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333 node DatanodeRegistration(127.0.0.1:39801, datanodeUuid=3b0f0fec-8b12-4bb0-9296-5c650e0a5410, infoPort=44799, infoSecurePort=0, ipcPort=35655, storageInfo=lv=-57;cid=testClusterID;nsid=204110191;c=1689656715810), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-18 05:05:17,869 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x75853e586a6b1d77: Processing first storage report for DS-ebdde9ea-77a3-4732-9998-b670c58d283a from datanode d4e5ff0e-b81a-4ef5-83b6-3e12d0570039 2023-07-18 05:05:17,869 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x75853e586a6b1d77: from storage DS-ebdde9ea-77a3-4732-9998-b670c58d283a node DatanodeRegistration(127.0.0.1:35281, datanodeUuid=d4e5ff0e-b81a-4ef5-83b6-3e12d0570039, infoPort=41481, infoSecurePort=0, ipcPort=46529, storageInfo=lv=-57;cid=testClusterID;nsid=204110191;c=1689656715810), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-18 05:05:17,869 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3bcf145430bfd1a7: Processing first storage report for DS-6a18feae-b927-4012-acb0-86692e6e88f7 from datanode 3b0f0fec-8b12-4bb0-9296-5c650e0a5410 2023-07-18 05:05:17,869 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3bcf145430bfd1a7: from storage DS-6a18feae-b927-4012-acb0-86692e6e88f7 node DatanodeRegistration(127.0.0.1:39801, datanodeUuid=3b0f0fec-8b12-4bb0-9296-5c650e0a5410, infoPort=44799, infoSecurePort=0, ipcPort=35655, storageInfo=lv=-57;cid=testClusterID;nsid=204110191;c=1689656715810), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-18 05:05:17,869 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x75853e586a6b1d77: Processing first storage report for DS-f9b65887-c028-408b-9aab-d24e19bed845 from datanode d4e5ff0e-b81a-4ef5-83b6-3e12d0570039 2023-07-18 05:05:17,869 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x75853e586a6b1d77: from storage DS-f9b65887-c028-408b-9aab-d24e19bed845 node DatanodeRegistration(127.0.0.1:35281, datanodeUuid=d4e5ff0e-b81a-4ef5-83b6-3e12d0570039, infoPort=41481, infoSecurePort=0, ipcPort=46529, storageInfo=lv=-57;cid=testClusterID;nsid=204110191;c=1689656715810), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-18 05:05:17,920 DEBUG [Listener at localhost/35655] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767 2023-07-18 05:05:17,980 INFO [Listener at localhost/35655] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/zookeeper_0, clientPort=52059, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-18 05:05:17,998 INFO [Listener at localhost/35655] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52059 2023-07-18 05:05:18,009 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:18,012 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:18,678 INFO [Listener at localhost/35655] util.FSUtils(471): Created version file at hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9 with version=8 2023-07-18 05:05:18,679 INFO [Listener at localhost/35655] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/hbase-staging 2023-07-18 05:05:19,022 INFO [Listener at localhost/35655] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-18 05:05:19,489 INFO [Listener at localhost/35655] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-18 05:05:19,522 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:19,522 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:19,522 INFO [Listener at localhost/35655] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-18 05:05:19,522 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:19,523 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-18 05:05:19,666 INFO [Listener at localhost/35655] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-18 05:05:19,743 DEBUG [Listener at localhost/35655] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-18 05:05:19,839 INFO [Listener at localhost/35655] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43045 2023-07-18 05:05:19,849 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:19,851 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:19,874 INFO [Listener at localhost/35655] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43045 connecting to ZooKeeper ensemble=127.0.0.1:52059 2023-07-18 05:05:19,917 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:430450x0, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-18 05:05:19,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:43045-0x10176d1f5910000 connected 2023-07-18 05:05:19,947 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-18 05:05:19,947 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 05:05:19,951 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-18 05:05:19,958 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43045 2023-07-18 05:05:19,959 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43045 2023-07-18 05:05:19,959 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43045 2023-07-18 05:05:19,959 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43045 2023-07-18 05:05:19,960 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43045 2023-07-18 05:05:19,966 INFO [Listener at localhost/35655] master.HMaster(444): hbase.rootdir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9, hbase.cluster.distributed=false 2023-07-18 05:05:20,035 INFO [Listener at localhost/35655] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-18 05:05:20,035 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:20,035 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:20,036 INFO [Listener at localhost/35655] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-18 05:05:20,036 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:20,036 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-18 05:05:20,041 INFO [Listener at localhost/35655] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-18 05:05:20,043 INFO [Listener at localhost/35655] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35379 2023-07-18 05:05:20,046 INFO [Listener at localhost/35655] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-18 05:05:20,051 DEBUG [Listener at localhost/35655] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-18 05:05:20,053 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:20,054 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:20,056 INFO [Listener at localhost/35655] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35379 connecting to ZooKeeper ensemble=127.0.0.1:52059 2023-07-18 05:05:20,059 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:353790x0, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-18 05:05:20,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:35379-0x10176d1f5910001 connected 2023-07-18 05:05:20,060 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-18 05:05:20,062 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 05:05:20,063 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-18 05:05:20,063 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35379 2023-07-18 05:05:20,063 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35379 2023-07-18 05:05:20,064 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35379 2023-07-18 05:05:20,064 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35379 2023-07-18 05:05:20,065 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35379 2023-07-18 05:05:20,079 INFO [Listener at localhost/35655] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-18 05:05:20,080 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:20,080 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:20,080 INFO [Listener at localhost/35655] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-18 05:05:20,081 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:20,081 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-18 05:05:20,081 INFO [Listener at localhost/35655] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-18 05:05:20,082 INFO [Listener at localhost/35655] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44031 2023-07-18 05:05:20,083 INFO [Listener at localhost/35655] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-18 05:05:20,084 DEBUG [Listener at localhost/35655] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-18 05:05:20,085 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:20,087 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:20,089 INFO [Listener at localhost/35655] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44031 connecting to ZooKeeper ensemble=127.0.0.1:52059 2023-07-18 05:05:20,093 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:440310x0, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-18 05:05:20,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:44031-0x10176d1f5910002 connected 2023-07-18 05:05:20,094 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-18 05:05:20,095 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 05:05:20,096 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-18 05:05:20,096 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44031 2023-07-18 05:05:20,097 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44031 2023-07-18 05:05:20,097 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44031 2023-07-18 05:05:20,098 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44031 2023-07-18 05:05:20,098 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44031 2023-07-18 05:05:20,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:20,111 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-18 05:05:20,113 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:20,131 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-18 05:05:20,131 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-18 05:05:20,131 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-18 05:05:20,131 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:20,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-18 05:05:20,134 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,43045,1689656718865 from backup master directory 2023-07-18 05:05:20,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-18 05:05:20,137 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:20,137 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-18 05:05:20,137 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-18 05:05:20,137 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:20,140 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-18 05:05:20,141 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-18 05:05:20,231 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/hbase.id with ID: 840b74ac-9460-422b-b113-cc18493fd4dc 2023-07-18 05:05:20,277 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:20,294 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:20,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x23337beb to 127.0.0.1:52059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 05:05:20,374 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13bda823, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 05:05:20,399 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-18 05:05:20,401 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-18 05:05:20,420 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-18 05:05:20,420 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-18 05:05:20,422 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-18 05:05:20,427 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-18 05:05:20,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 05:05:20,463 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/data/master/store-tmp 2023-07-18 05:05:20,504 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:20,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-18 05:05:20,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 05:05:20,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 05:05:20,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-18 05:05:20,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 05:05:20,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 05:05:20,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-18 05:05:20,507 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/WALs/jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:20,527 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43045%2C1689656718865, suffix=, logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/WALs/jenkins-hbase4.apache.org,43045,1689656718865, archiveDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/oldWALs, maxLogs=10 2023-07-18 05:05:20,580 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK] 2023-07-18 05:05:20,580 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK] 2023-07-18 05:05:20,588 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 05:05:20,655 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/WALs/jenkins-hbase4.apache.org,43045,1689656718865/jenkins-hbase4.apache.org%2C43045%2C1689656718865.1689656720536 2023-07-18 05:05:20,655 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK], DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK]] 2023-07-18 05:05:20,656 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:20,656 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:20,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-18 05:05:20,662 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-18 05:05:20,717 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-18 05:05:20,724 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-18 05:05:20,748 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-18 05:05:20,760 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:20,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-18 05:05:20,769 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-18 05:05:20,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-18 05:05:20,787 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-18 05:05:20,788 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74740805, jitterRate=0.11372478306293488}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 05:05:20,788 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-18 05:05:20,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-18 05:05:20,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-18 05:05:20,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-18 05:05:20,813 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-18 05:05:20,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-18 05:05:20,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 34 msec 2023-07-18 05:05:20,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-18 05:05:20,878 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-18 05:05:20,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-18 05:05:20,909 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-18 05:05:20,913 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-18 05:05:20,915 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-18 05:05:20,920 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-18 05:05:20,925 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-18 05:05:20,930 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:20,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-18 05:05:20,932 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-18 05:05:20,945 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-18 05:05:20,950 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-18 05:05:20,950 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-18 05:05:20,951 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:20,951 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-18 05:05:20,955 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,43045,1689656718865, sessionid=0x10176d1f5910000, setting cluster-up flag (Was=false) 2023-07-18 05:05:20,975 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:20,980 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-18 05:05:20,982 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:20,986 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:20,992 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-18 05:05:20,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:20,996 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.hbase-snapshot/.tmp 2023-07-18 05:05:21,001 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(951): ClusterId : 840b74ac-9460-422b-b113-cc18493fd4dc 2023-07-18 05:05:21,001 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(951): ClusterId : 840b74ac-9460-422b-b113-cc18493fd4dc 2023-07-18 05:05:21,006 DEBUG [RS:1;jenkins-hbase4:44031] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-18 05:05:21,006 DEBUG [RS:0;jenkins-hbase4:35379] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-18 05:05:21,011 DEBUG [RS:0;jenkins-hbase4:35379] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-18 05:05:21,011 DEBUG [RS:1;jenkins-hbase4:44031] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-18 05:05:21,011 DEBUG [RS:0;jenkins-hbase4:35379] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-18 05:05:21,011 DEBUG [RS:1;jenkins-hbase4:44031] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-18 05:05:21,016 DEBUG [RS:0;jenkins-hbase4:35379] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-18 05:05:21,016 DEBUG [RS:1;jenkins-hbase4:44031] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-18 05:05:21,017 DEBUG [RS:0;jenkins-hbase4:35379] zookeeper.ReadOnlyZKClient(139): Connect 0x70298017 to 127.0.0.1:52059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 05:05:21,017 DEBUG [RS:1;jenkins-hbase4:44031] zookeeper.ReadOnlyZKClient(139): Connect 0x446fc840 to 127.0.0.1:52059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 05:05:21,024 DEBUG [RS:0;jenkins-hbase4:35379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eee7062, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 05:05:21,024 DEBUG [RS:1;jenkins-hbase4:44031] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4aa83ebd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 05:05:21,024 DEBUG [RS:0;jenkins-hbase4:35379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ad1eb67, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-18 05:05:21,024 DEBUG [RS:1;jenkins-hbase4:44031] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e97a330, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-18 05:05:21,047 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:44031 2023-07-18 05:05:21,048 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35379 2023-07-18 05:05:21,051 INFO [RS:1;jenkins-hbase4:44031] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-18 05:05:21,051 INFO [RS:0;jenkins-hbase4:35379] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-18 05:05:21,051 INFO [RS:0;jenkins-hbase4:35379] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-18 05:05:21,051 INFO [RS:1;jenkins-hbase4:44031] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-18 05:05:21,052 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1022): About to register with Master. 2023-07-18 05:05:21,052 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1022): About to register with Master. 2023-07-18 05:05:21,054 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,43045,1689656718865 with isa=jenkins-hbase4.apache.org/172.31.14.131:44031, startcode=1689656720079 2023-07-18 05:05:21,054 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,43045,1689656718865 with isa=jenkins-hbase4.apache.org/172.31.14.131:35379, startcode=1689656720034 2023-07-18 05:05:21,073 DEBUG [RS:1;jenkins-hbase4:44031] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-18 05:05:21,073 DEBUG [RS:0;jenkins-hbase4:35379] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-18 05:05:21,128 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-18 05:05:21,134 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60139, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-18 05:05:21,134 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42299, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-18 05:05:21,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-18 05:05:21,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-18 05:05:21,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-18 05:05:21,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-18 05:05:21,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-18 05:05:21,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-18 05:05:21,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,140 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689656751140 2023-07-18 05:05:21,143 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-18 05:05:21,145 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43045] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 05:05:21,148 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-18 05:05:21,149 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-18 05:05:21,153 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-18 05:05:21,154 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-18 05:05:21,155 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43045] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 05:05:21,163 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-18 05:05:21,163 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-18 05:05:21,163 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-18 05:05:21,163 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-18 05:05:21,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-18 05:05:21,166 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-18 05:05:21,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-18 05:05:21,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-18 05:05:21,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-18 05:05:21,171 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689656721171,5,FailOnTimeoutGroup] 2023-07-18 05:05:21,172 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689656721172,5,FailOnTimeoutGroup] 2023-07-18 05:05:21,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-18 05:05:21,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,177 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(2830): Master is not running yet 2023-07-18 05:05:21,177 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(2830): Master is not running yet 2023-07-18 05:05:21,177 WARN [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-18 05:05:21,177 WARN [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-18 05:05:21,194 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-18 05:05:21,195 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-18 05:05:21,195 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9 2023-07-18 05:05:21,212 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:21,214 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-18 05:05:21,217 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info 2023-07-18 05:05:21,217 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-18 05:05:21,219 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:21,219 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-18 05:05:21,221 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 05:05:21,222 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-18 05:05:21,223 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:21,223 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-18 05:05:21,225 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table 2023-07-18 05:05:21,225 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-18 05:05:21,226 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:21,228 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740 2023-07-18 05:05:21,228 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740 2023-07-18 05:05:21,232 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-18 05:05:21,234 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-18 05:05:21,238 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-18 05:05:21,239 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64526978, jitterRate=-0.03847309947013855}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-18 05:05:21,239 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-18 05:05:21,239 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-18 05:05:21,239 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-18 05:05:21,239 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-18 05:05:21,239 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-18 05:05:21,239 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-18 05:05:21,240 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-18 05:05:21,240 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-18 05:05:21,246 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-18 05:05:21,246 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-18 05:05:21,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-18 05:05:21,267 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-18 05:05:21,270 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-18 05:05:21,278 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,43045,1689656718865 with isa=jenkins-hbase4.apache.org/172.31.14.131:44031, startcode=1689656720079 2023-07-18 05:05:21,278 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,43045,1689656718865 with isa=jenkins-hbase4.apache.org/172.31.14.131:35379, startcode=1689656720034 2023-07-18 05:05:21,284 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43045] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:21,290 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43045] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:21,291 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9 2023-07-18 05:05:21,291 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:36887 2023-07-18 05:05:21,292 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-18 05:05:21,292 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9 2023-07-18 05:05:21,292 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:36887 2023-07-18 05:05:21,292 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-18 05:05:21,301 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:21,302 DEBUG [RS:1;jenkins-hbase4:44031] zookeeper.ZKUtil(162): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:21,302 DEBUG [RS:0;jenkins-hbase4:35379] zookeeper.ZKUtil(162): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:21,302 WARN [RS:1;jenkins-hbase4:44031] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-18 05:05:21,303 INFO [RS:1;jenkins-hbase4:44031] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 05:05:21,303 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1948): logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:21,302 WARN [RS:0;jenkins-hbase4:35379] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-18 05:05:21,305 INFO [RS:0;jenkins-hbase4:35379] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 05:05:21,306 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1948): logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:21,307 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44031,1689656720079] 2023-07-18 05:05:21,307 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35379,1689656720034] 2023-07-18 05:05:21,314 DEBUG [RS:1;jenkins-hbase4:44031] zookeeper.ZKUtil(162): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:21,314 DEBUG [RS:0;jenkins-hbase4:35379] zookeeper.ZKUtil(162): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:21,315 DEBUG [RS:1;jenkins-hbase4:44031] zookeeper.ZKUtil(162): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:21,315 DEBUG [RS:0;jenkins-hbase4:35379] zookeeper.ZKUtil(162): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:21,325 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-18 05:05:21,325 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-18 05:05:21,334 INFO [RS:1;jenkins-hbase4:44031] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-18 05:05:21,334 INFO [RS:0;jenkins-hbase4:35379] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-18 05:05:21,354 INFO [RS:1;jenkins-hbase4:44031] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-18 05:05:21,354 INFO [RS:0;jenkins-hbase4:35379] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-18 05:05:21,360 INFO [RS:1;jenkins-hbase4:44031] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-18 05:05:21,360 INFO [RS:0;jenkins-hbase4:35379] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-18 05:05:21,361 INFO [RS:1;jenkins-hbase4:44031] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,361 INFO [RS:0;jenkins-hbase4:35379] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,362 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-18 05:05:21,362 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-18 05:05:21,369 INFO [RS:0;jenkins-hbase4:35379] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,369 INFO [RS:1;jenkins-hbase4:44031] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,369 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,369 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,369 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,369 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-18 05:05:21,370 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-18 05:05:21,370 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,370 DEBUG [RS:0;jenkins-hbase4:35379] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,371 DEBUG [RS:1;jenkins-hbase4:44031] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:21,373 INFO [RS:1;jenkins-hbase4:44031] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,373 INFO [RS:0;jenkins-hbase4:35379] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,373 INFO [RS:1;jenkins-hbase4:44031] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,373 INFO [RS:0;jenkins-hbase4:35379] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,373 INFO [RS:1;jenkins-hbase4:44031] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,373 INFO [RS:0;jenkins-hbase4:35379] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,389 INFO [RS:1;jenkins-hbase4:44031] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-18 05:05:21,389 INFO [RS:0;jenkins-hbase4:35379] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-18 05:05:21,393 INFO [RS:1;jenkins-hbase4:44031] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44031,1689656720079-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,393 INFO [RS:0;jenkins-hbase4:35379] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35379,1689656720034-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,415 INFO [RS:0;jenkins-hbase4:35379] regionserver.Replication(203): jenkins-hbase4.apache.org,35379,1689656720034 started 2023-07-18 05:05:21,415 INFO [RS:1;jenkins-hbase4:44031] regionserver.Replication(203): jenkins-hbase4.apache.org,44031,1689656720079 started 2023-07-18 05:05:21,415 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,35379,1689656720034, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35379, sessionid=0x10176d1f5910001 2023-07-18 05:05:21,416 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,44031,1689656720079, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44031, sessionid=0x10176d1f5910002 2023-07-18 05:05:21,416 DEBUG [RS:1;jenkins-hbase4:44031] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-18 05:05:21,416 DEBUG [RS:0;jenkins-hbase4:35379] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-18 05:05:21,416 DEBUG [RS:1;jenkins-hbase4:44031] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:21,416 DEBUG [RS:0;jenkins-hbase4:35379] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:21,416 DEBUG [RS:1;jenkins-hbase4:44031] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44031,1689656720079' 2023-07-18 05:05:21,416 DEBUG [RS:0;jenkins-hbase4:35379] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35379,1689656720034' 2023-07-18 05:05:21,417 DEBUG [RS:0;jenkins-hbase4:35379] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-18 05:05:21,417 DEBUG [RS:1;jenkins-hbase4:44031] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-18 05:05:21,417 DEBUG [RS:1;jenkins-hbase4:44031] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-18 05:05:21,418 DEBUG [RS:0;jenkins-hbase4:35379] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-18 05:05:21,418 DEBUG [RS:1;jenkins-hbase4:44031] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-18 05:05:21,418 DEBUG [RS:0;jenkins-hbase4:35379] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-18 05:05:21,418 DEBUG [RS:1;jenkins-hbase4:44031] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-18 05:05:21,418 DEBUG [RS:0;jenkins-hbase4:35379] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-18 05:05:21,418 DEBUG [RS:1;jenkins-hbase4:44031] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:21,419 DEBUG [RS:0;jenkins-hbase4:35379] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:21,419 DEBUG [RS:1;jenkins-hbase4:44031] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44031,1689656720079' 2023-07-18 05:05:21,419 DEBUG [RS:1;jenkins-hbase4:44031] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-18 05:05:21,419 DEBUG [RS:0;jenkins-hbase4:35379] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35379,1689656720034' 2023-07-18 05:05:21,419 DEBUG [RS:0;jenkins-hbase4:35379] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-18 05:05:21,420 DEBUG [RS:0;jenkins-hbase4:35379] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-18 05:05:21,420 DEBUG [RS:1;jenkins-hbase4:44031] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-18 05:05:21,420 DEBUG [RS:0;jenkins-hbase4:35379] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-18 05:05:21,420 DEBUG [RS:1;jenkins-hbase4:44031] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-18 05:05:21,420 INFO [RS:0;jenkins-hbase4:35379] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-18 05:05:21,420 INFO [RS:1;jenkins-hbase4:44031] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-18 05:05:21,421 INFO [RS:0;jenkins-hbase4:35379] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-18 05:05:21,421 INFO [RS:1;jenkins-hbase4:44031] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-18 05:05:21,423 DEBUG [jenkins-hbase4:43045] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-18 05:05:21,428 DEBUG [jenkins-hbase4:43045] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-18 05:05:21,434 DEBUG [jenkins-hbase4:43045] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:05:21,434 DEBUG [jenkins-hbase4:43045] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-18 05:05:21,434 DEBUG [jenkins-hbase4:43045] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-18 05:05:21,437 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44031,1689656720079, state=OPENING 2023-07-18 05:05:21,444 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-18 05:05:21,445 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:21,446 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 05:05:21,449 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44031,1689656720079}] 2023-07-18 05:05:21,533 INFO [RS:1;jenkins-hbase4:44031] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44031%2C1689656720079, suffix=, logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079, archiveDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs, maxLogs=32 2023-07-18 05:05:21,533 INFO [RS:0;jenkins-hbase4:35379] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35379%2C1689656720034, suffix=, logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034, archiveDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs, maxLogs=32 2023-07-18 05:05:21,561 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK] 2023-07-18 05:05:21,561 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK] 2023-07-18 05:05:21,561 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK] 2023-07-18 05:05:21,562 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK] 2023-07-18 05:05:21,572 INFO [RS:1;jenkins-hbase4:44031] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540 2023-07-18 05:05:21,572 INFO [RS:0;jenkins-hbase4:35379] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540 2023-07-18 05:05:21,572 DEBUG [RS:1;jenkins-hbase4:44031] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK], DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK]] 2023-07-18 05:05:21,572 DEBUG [RS:0;jenkins-hbase4:35379] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK], DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK]] 2023-07-18 05:05:21,632 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:21,635 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-18 05:05:21,638 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33288, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-18 05:05:21,649 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-18 05:05:21,649 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 05:05:21,653 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta, suffix=.meta, logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079, archiveDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs, maxLogs=32 2023-07-18 05:05:21,670 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK] 2023-07-18 05:05:21,670 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK] 2023-07-18 05:05:21,678 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta 2023-07-18 05:05:21,681 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK], DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK]] 2023-07-18 05:05:21,681 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:21,683 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-18 05:05:21,707 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-18 05:05:21,740 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-18 05:05:21,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-18 05:05:21,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:21,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-18 05:05:21,749 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-18 05:05:21,752 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-18 05:05:21,754 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info 2023-07-18 05:05:21,754 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info 2023-07-18 05:05:21,754 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-18 05:05:21,755 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:21,756 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-18 05:05:21,757 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 05:05:21,757 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 05:05:21,758 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-18 05:05:21,758 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:21,759 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-18 05:05:21,760 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table 2023-07-18 05:05:21,760 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table 2023-07-18 05:05:21,761 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-18 05:05:21,761 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:21,763 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740 2023-07-18 05:05:21,766 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740 2023-07-18 05:05:21,770 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-18 05:05:21,772 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-18 05:05:21,773 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59150187, jitterRate=-0.11859352886676788}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-18 05:05:21,773 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-18 05:05:21,783 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689656721625 2023-07-18 05:05:21,803 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-18 05:05:21,804 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-18 05:05:21,805 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44031,1689656720079, state=OPEN 2023-07-18 05:05:21,808 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 05:05:21,808 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 05:05:21,812 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-18 05:05:21,813 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44031,1689656720079 in 359 msec 2023-07-18 05:05:21,817 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-18 05:05:21,818 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 559 msec 2023-07-18 05:05:21,822 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 773 msec 2023-07-18 05:05:21,823 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689656721822, completionTime=-1 2023-07-18 05:05:21,823 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-18 05:05:21,823 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-18 05:05:21,896 DEBUG [hconnection-0x7ba23e29-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:21,899 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33296, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:21,917 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-18 05:05:21,917 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689656781917 2023-07-18 05:05:21,917 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689656841917 2023-07-18 05:05:21,917 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 93 msec 2023-07-18 05:05:21,939 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43045,1689656718865-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,940 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43045,1689656718865-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,940 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43045,1689656718865-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,942 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:43045, period=300000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,942 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:21,948 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-18 05:05:21,956 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-18 05:05:21,958 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-18 05:05:21,968 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-18 05:05:21,970 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-18 05:05:21,973 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-18 05:05:21,998 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:22,000 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529 empty. 2023-07-18 05:05:22,001 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:22,002 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-18 05:05:22,034 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-18 05:05:22,037 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 01bbc36252aecc1c98ab9ce391c47529, NAME => 'hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp 2023-07-18 05:05:22,054 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:22,054 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 01bbc36252aecc1c98ab9ce391c47529, disabling compactions & flushes 2023-07-18 05:05:22,054 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:22,054 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:22,054 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. after waiting 0 ms 2023-07-18 05:05:22,054 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:22,054 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:22,054 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 01bbc36252aecc1c98ab9ce391c47529: 2023-07-18 05:05:22,058 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-18 05:05:22,073 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689656722061"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689656722061"}]},"ts":"1689656722061"} 2023-07-18 05:05:22,103 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-18 05:05:22,105 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-18 05:05:22,111 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689656722106"}]},"ts":"1689656722106"} 2023-07-18 05:05:22,116 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-18 05:05:22,121 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-18 05:05:22,122 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:05:22,122 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-18 05:05:22,122 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-18 05:05:22,126 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN}] 2023-07-18 05:05:22,129 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN 2023-07-18 05:05:22,130 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44031,1689656720079; forceNewPlan=false, retain=false 2023-07-18 05:05:22,284 INFO [jenkins-hbase4:43045] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-18 05:05:22,285 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=01bbc36252aecc1c98ab9ce391c47529, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:22,286 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689656722285"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689656722285"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689656722285"}]},"ts":"1689656722285"} 2023-07-18 05:05:22,294 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 01bbc36252aecc1c98ab9ce391c47529, server=jenkins-hbase4.apache.org,44031,1689656720079}] 2023-07-18 05:05:22,455 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:22,456 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 01bbc36252aecc1c98ab9ce391c47529, NAME => 'hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:22,456 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:22,457 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:22,457 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:22,457 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:22,459 INFO [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:22,462 DEBUG [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/info 2023-07-18 05:05:22,462 DEBUG [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/info 2023-07-18 05:05:22,462 INFO [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01bbc36252aecc1c98ab9ce391c47529 columnFamilyName info 2023-07-18 05:05:22,463 INFO [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] regionserver.HStore(310): Store=01bbc36252aecc1c98ab9ce391c47529/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:22,465 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:22,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:22,471 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:22,474 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-18 05:05:22,475 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 01bbc36252aecc1c98ab9ce391c47529; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67364413, jitterRate=0.0038079768419265747}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 05:05:22,475 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 01bbc36252aecc1c98ab9ce391c47529: 2023-07-18 05:05:22,478 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529., pid=6, masterSystemTime=1689656722448 2023-07-18 05:05:22,482 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:22,483 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:22,484 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=01bbc36252aecc1c98ab9ce391c47529, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:22,484 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689656722483"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689656722483"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689656722483"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689656722483"}]},"ts":"1689656722483"} 2023-07-18 05:05:22,493 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-18 05:05:22,493 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 01bbc36252aecc1c98ab9ce391c47529, server=jenkins-hbase4.apache.org,44031,1689656720079 in 195 msec 2023-07-18 05:05:22,497 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-18 05:05:22,498 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN in 368 msec 2023-07-18 05:05:22,499 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-18 05:05:22,499 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689656722499"}]},"ts":"1689656722499"} 2023-07-18 05:05:22,502 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-18 05:05:22,505 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-18 05:05:22,508 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 546 msec 2023-07-18 05:05:22,570 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-18 05:05:22,572 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-18 05:05:22,572 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:22,611 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-18 05:05:22,634 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-18 05:05:22,642 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 41 msec 2023-07-18 05:05:22,656 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-18 05:05:22,668 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-18 05:05:22,675 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-07-18 05:05:22,682 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-18 05:05:22,686 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-18 05:05:22,686 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.548sec 2023-07-18 05:05:22,689 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-18 05:05:22,691 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-18 05:05:22,691 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-18 05:05:22,692 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43045,1689656718865-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-18 05:05:22,692 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43045,1689656718865-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-18 05:05:22,701 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-18 05:05:22,707 DEBUG [Listener at localhost/35655] zookeeper.ReadOnlyZKClient(139): Connect 0x044a9743 to 127.0.0.1:52059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 05:05:22,712 DEBUG [Listener at localhost/35655] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ca72c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 05:05:22,751 DEBUG [hconnection-0x32508046-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:22,766 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33300, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:22,776 INFO [Listener at localhost/35655] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:22,789 DEBUG [Listener at localhost/35655] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-18 05:05:22,793 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-18 05:05:22,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-18 05:05:22,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-18 05:05:22,818 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-18 05:05:22,820 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-18 05:05:22,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-18 05:05:22,827 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:22,828 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1 empty. 2023-07-18 05:05:22,830 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:22,830 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-18 05:05:22,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-18 05:05:22,864 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-18 05:05:22,872 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2bc2fc351e77525b9523e5bc259c0fe1, NAME => 't,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/.tmp 2023-07-18 05:05:22,906 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:22,906 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 2bc2fc351e77525b9523e5bc259c0fe1, disabling compactions & flushes 2023-07-18 05:05:22,906 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:22,906 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:22,906 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. after waiting 0 ms 2023-07-18 05:05:22,907 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:22,907 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:22,907 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 2bc2fc351e77525b9523e5bc259c0fe1: 2023-07-18 05:05:22,913 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-18 05:05:22,915 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689656722915"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689656722915"}]},"ts":"1689656722915"} 2023-07-18 05:05:22,918 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-18 05:05:22,920 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-18 05:05:22,920 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689656722920"}]},"ts":"1689656722920"} 2023-07-18 05:05:22,923 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-18 05:05:22,929 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-18 05:05:22,930 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 05:05:22,930 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-18 05:05:22,930 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-18 05:05:22,930 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN}] 2023-07-18 05:05:22,933 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN 2023-07-18 05:05:22,935 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44031,1689656720079; forceNewPlan=false, retain=false 2023-07-18 05:05:22,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-18 05:05:23,085 INFO [jenkins-hbase4:43045] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-18 05:05:23,086 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2bc2fc351e77525b9523e5bc259c0fe1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:23,087 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689656723086"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689656723086"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689656723086"}]},"ts":"1689656723086"} 2023-07-18 05:05:23,091 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 2bc2fc351e77525b9523e5bc259c0fe1, server=jenkins-hbase4.apache.org,44031,1689656720079}] 2023-07-18 05:05:23,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-18 05:05:23,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:23,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2bc2fc351e77525b9523e5bc259c0fe1, NAME => 't,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:23,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:23,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:23,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:23,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:23,253 INFO [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:23,256 DEBUG [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/f 2023-07-18 05:05:23,256 DEBUG [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/f 2023-07-18 05:05:23,256 INFO [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2bc2fc351e77525b9523e5bc259c0fe1 columnFamilyName f 2023-07-18 05:05:23,257 INFO [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] regionserver.HStore(310): Store=2bc2fc351e77525b9523e5bc259c0fe1/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:23,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:23,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:23,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:23,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-18 05:05:23,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2bc2fc351e77525b9523e5bc259c0fe1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72280152, jitterRate=0.07705819606781006}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 05:05:23,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2bc2fc351e77525b9523e5bc259c0fe1: 2023-07-18 05:05:23,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1., pid=11, masterSystemTime=1689656723245 2023-07-18 05:05:23,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:23,271 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:23,272 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2bc2fc351e77525b9523e5bc259c0fe1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:23,272 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689656723272"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689656723272"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689656723272"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689656723272"}]},"ts":"1689656723272"} 2023-07-18 05:05:23,278 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-18 05:05:23,278 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 2bc2fc351e77525b9523e5bc259c0fe1, server=jenkins-hbase4.apache.org,44031,1689656720079 in 184 msec 2023-07-18 05:05:23,281 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-18 05:05:23,281 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN in 348 msec 2023-07-18 05:05:23,282 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-18 05:05:23,282 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689656723282"}]},"ts":"1689656723282"} 2023-07-18 05:05:23,284 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-18 05:05:23,287 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-18 05:05:23,289 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 475 msec 2023-07-18 05:05:23,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-18 05:05:23,450 INFO [Listener at localhost/35655] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-18 05:05:23,450 DEBUG [Listener at localhost/35655] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-18 05:05:23,452 INFO [Listener at localhost/35655] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-18 05:05:23,457 INFO [Listener at localhost/35655] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-18 05:05:23,458 INFO [Listener at localhost/35655] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-18 05:05:23,458 INFO [Listener at localhost/35655] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-18 05:05:23,504 INFO [Listener at localhost/35655] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=291, OpenFileDescriptor=581, MaxFileDescriptor=60000, SystemLoadAverage=152, ProcessCount=170, AvailableMemoryMB=6111 2023-07-18 05:05:23,505 INFO [Listener at localhost/35655] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-18 05:05:23,509 DEBUG [Listener at localhost/35655] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-18 05:05:23,510 DEBUG [Listener at localhost/35655] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:23,510 INFO [Listener at localhost/35655] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-18 05:05:23,510 INFO [Listener at localhost/35655] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-18 05:05:23,513 INFO [Listener at localhost/35655] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689656722499/Put/vlen=2/seqid=0} 2023-07-18 05:05:23,513 INFO [Listener at localhost/35655] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-18 05:05:23,513 INFO [Listener at localhost/35655] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529./info:regioninfo/1689656722483/Put/vlen=41/seqid=0, hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529./info:seqnumDuringOpen/1689656722483/Put/vlen=8/seqid=0, hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529./info:server/1689656722483/Put/vlen=31/seqid=0, hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529./info:serverstartcode/1689656722483/Put/vlen=8/seqid=0, hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529./info:sn/1689656722285/Put/vlen=45/seqid=0, hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529./info:state/1689656722483/Put/vlen=4/seqid=0} 2023-07-18 05:05:23,513 INFO [Listener at localhost/35655] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 01bbc36252aecc1c98ab9ce391c47529, NAME => 'hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:23,513 INFO [Listener at localhost/35655] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689656723282/Put/vlen=2/seqid=0} 2023-07-18 05:05:23,514 INFO [Listener at localhost/35655] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-18 05:05:23,514 INFO [Listener at localhost/35655] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1./info:regioninfo/1689656723272/Put/vlen=35/seqid=0, t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1./info:seqnumDuringOpen/1689656723272/Put/vlen=8/seqid=0, t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1./info:server/1689656723272/Put/vlen=31/seqid=0, t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1./info:serverstartcode/1689656723272/Put/vlen=8/seqid=0, t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1./info:sn/1689656723086/Put/vlen=45/seqid=0, t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1./info:state/1689656723272/Put/vlen=4/seqid=0} 2023-07-18 05:05:23,514 INFO [Listener at localhost/35655] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 2bc2fc351e77525b9523e5bc259c0fe1, NAME => 't,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:23,515 DEBUG [Listener at localhost/35655] zookeeper.ReadOnlyZKClient(139): Connect 0x0e6a4777 to 127.0.0.1:52059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 05:05:23,520 DEBUG [Listener at localhost/35655] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c11e80b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 05:05:23,520 INFO [Listener at localhost/35655] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-18 05:05:23,520 INFO [Listener at localhost/35655] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-18 05:05:23,522 DEBUG [hconnection-0x7944646-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:23,524 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:23,529 INFO [Listener at localhost/35655] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-18 05:05:23,532 ERROR [Listener at localhost/35655] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,44031,1689656720079: die! ***** 2023-07-18 05:05:23,532 ERROR [Listener at localhost/35655] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-18 05:05:23,535 DEBUG [Listener at localhost/35655] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-18 05:05:23,538 DEBUG [Listener at localhost/35655] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-18 05:05:23,539 DEBUG [Listener at localhost/35655] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-18 05:05:23,539 DEBUG [Listener at localhost/35655] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-18 05:05:23,540 INFO [Listener at localhost/35655] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 991952896, "init": 513802240, "max": 2051014656, "used": 174261496 }, "NonHeapMemoryUsage": { "committed": 112877568, "init": 2555904, "max": -1, "used": 110875920 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-18 05:05:23,545 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43045] master.MasterRpcServices(612): jenkins-hbase4.apache.org,44031,1689656720079 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,44031,1689656720079: die! ***** 2023-07-18 05:05:23,547 INFO [Listener at localhost/35655] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,44031,1689656720079' ***** 2023-07-18 05:05:23,547 INFO [Listener at localhost/35655] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-18 05:05:23,548 INFO [RS:1;jenkins-hbase4:44031] regionserver.HeapMemoryManager(220): Stopping 2023-07-18 05:05:23,548 INFO [RS:1;jenkins-hbase4:44031] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-18 05:05:23,548 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-18 05:05:23,548 INFO [RS:1;jenkins-hbase4:44031] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-18 05:05:23,548 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(3305): Received CLOSE for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44031] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 7044040343066509324 2023-07-18 05:05:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44031] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 05:05:23,549 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(3305): Received CLOSE for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44031] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:33314 deadline: 1689656783548, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting 2023-07-18 05:05:23,549 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:23,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2bc2fc351e77525b9523e5bc259c0fe1, disabling compactions & flushes 2023-07-18 05:05:23,550 DEBUG [RS:1;jenkins-hbase4:44031] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x446fc840 to 127.0.0.1:52059 2023-07-18 05:05:23,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:23,550 DEBUG [RS:1;jenkins-hbase4:44031] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:23,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:23,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. after waiting 0 ms 2023-07-18 05:05:23,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:23,550 INFO [RS:1;jenkins-hbase4:44031] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-18 05:05:23,550 INFO [RS:1;jenkins-hbase4:44031] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-18 05:05:23,550 INFO [RS:1;jenkins-hbase4:44031] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-18 05:05:23,550 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-18 05:05:23,551 DEBUG [Listener at localhost/35655] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause 2023-07-18 05:05:23,552 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-18 05:05:23,551 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) 2023-07-18 05:05:23,551 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1474): Waiting on 3 regions to close at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-18 05:05:23,552 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 05:05:23,553 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more 2023-07-18 05:05:23,552 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1478): Online Regions={2bc2fc351e77525b9523e5bc259c0fe1=t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1., 1588230740=hbase:meta,,1.1588230740, 01bbc36252aecc1c98ab9ce391c47529=hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.} Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) 2023-07-18 05:05:23,553 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) 2023-07-18 05:05:23,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) 2023-07-18 05:05:23,553 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) 2023-07-18 05:05:23,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2bc2fc351e77525b9523e5bc259c0fe1: at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-18 05:05:23,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:23,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 01bbc36252aecc1c98ab9ce391c47529, disabling compactions & flushes 2023-07-18 05:05:23,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:23,554 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-18 05:05:23,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:23,555 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-18 05:05:23,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. after waiting 0 ms 2023-07-18 05:05:23,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:23,555 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-18 05:05:23,555 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-18 05:05:23,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44031] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 05:05:23,555 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-18 05:05:23,556 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:23,556 DEBUG [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1504): Waiting on 01bbc36252aecc1c98ab9ce391c47529, 1588230740 2023-07-18 05:05:23,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44031] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:33314 deadline: 1689656783555, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting 2023-07-18 05:05:23,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:23,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 01bbc36252aecc1c98ab9ce391c47529: 2023-07-18 05:05:23,557 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:23,660 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44031] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 05:05:23,660 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44031] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:33314 deadline: 1689656783660, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting 2023-07-18 05:05:23,756 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,44031,1689656720079; all regions closed. 2023-07-18 05:05:23,765 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=44031] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 05:05:23,766 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=44031] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:33314 deadline: 1689656783765, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44031,1689656720079 aborting 2023-07-18 05:05:23,770 DEBUG [RS:1;jenkins-hbase4:44031] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:23,771 INFO [RS:1;jenkins-hbase4:44031] regionserver.LeaseManager(133): Closed leases 2023-07-18 05:05:23,772 INFO [RS:1;jenkins-hbase4:44031] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-18 05:05:23,772 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-18 05:05:23,773 INFO [RS:1;jenkins-hbase4:44031] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44031 2023-07-18 05:05:23,781 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:23,781 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:23,781 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:23,781 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:23,782 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:23,783 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44031,1689656720079] 2023-07-18 05:05:23,784 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44031,1689656720079; numProcessing=1 2023-07-18 05:05:23,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:23,785 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,44031,1689656720079 znode expired, triggering replicatorRemoved event 2023-07-18 05:05:23,785 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44031,1689656720079 already deleted, retry=false 2023-07-18 05:05:23,785 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,44031,1689656720079 on jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:23,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:23,793 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,44031,1689656720079, splitWal=true, meta=true 2023-07-18 05:05:23,793 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,44031,1689656720079 (carryingMeta=true) jenkins-hbase4.apache.org,44031,1689656720079/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@5dfff4e[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-18 05:05:23,796 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,44031,1689656720079, splitWal=true, meta=true 2023-07-18 05:05:23,797 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,44031,1689656720079, splitWal=true, meta=true, isMeta: true 2023-07-18 05:05:23,799 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting 2023-07-18 05:05:23,800 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,44031,1689656720079 WAL count=1, meta=true 2023-07-18 05:05:23,804 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta}] 2023-07-18 05:05:23,812 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:23,814 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta, worker=jenkins-hbase4.apache.org,35379,1689656720034}] 2023-07-18 05:05:23,884 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 05:05:23,884 INFO [RS:1;jenkins-hbase4:44031] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,44031,1689656720079; zookeeper connection closed. 2023-07-18 05:05:23,884 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:44031-0x10176d1f5910002, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 05:05:23,885 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@78908d0b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@78908d0b 2023-07-18 05:05:23,971 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:44031 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:44031 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 05:05:23,972 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:44031 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:44031 2023-07-18 05:05:23,977 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:23,977 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-18 05:05:23,980 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-18 05:05:23,981 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35379] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-18 05:05:24,002 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta, size=3.1 K (3185bytes) 2023-07-18 05:05:24,003 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta 2023-07-18 05:05:24,008 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta after 5ms 2023-07-18 05:05:24,013 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta took 12ms 2023-07-18 05:05:24,032 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-18 05:05:24,034 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta so closing down 2023-07-18 05:05:24,034 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-18 05:05:24,040 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-18 05:05:24,043 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta.temp 2023-07-18 05:05:24,044 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-18 05:05:24,057 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-18 05:05:24,059 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta.temp to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-18 05:05:24,059 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 45 ms; skipped=1; WAL=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-18 05:05:24,060 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta, journal: Splitting hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta, size=3.1 K (3185bytes) at 1689656724002Finishing writing output for hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta so closing down at 1689656724034 (+32 ms)Creating recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta.temp at 1689656724043 (+9 ms)3 split writer threads finished at 1689656724044 (+1 ms)Closed recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689656724057 (+13 ms)Rename recovered edits hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta.temp to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689656724059 (+2 ms)Processed 11 edits across 1 Regions in 45 ms; skipped=1; WAL=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689656724059 2023-07-18 05:05:24,060 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta 2023-07-18 05:05:24,062 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-18 05:05:24,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.HMaster(3641): Remote procedure done, pid=14 2023-07-18 05:05:24,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] wal.WALSplitUtil(141): Moved hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs 2023-07-18 05:05:24,076 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-18 05:05:24,076 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta, worker=jenkins-hbase4.apache.org,35379,1689656720034 in 259 msec 2023-07-18 05:05:24,077 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:24,081 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-18 05:05:24,081 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C44031%2C1689656720079.meta.1689656721654.meta, worker=jenkins-hbase4.apache.org,35379,1689656720034 in 275 msec 2023-07-18 05:05:24,084 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting dir is empty, no logs to split. 2023-07-18 05:05:24,084 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,44031,1689656720079 WAL count=0, meta=true 2023-07-18 05:05:24,084 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,44031,1689656720079 WAL splitting is done? wals=0, meta=true 2023-07-18 05:05:24,085 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-18 05:05:24,086 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-18 05:05:24,087 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 05:05:24,238 DEBUG [jenkins-hbase4:43045] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-18 05:05:24,238 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35379,1689656720034, state=OPENING 2023-07-18 05:05:24,241 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 05:05:24,241 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 05:05:24,241 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35379,1689656720034}] 2023-07-18 05:05:24,274 WARN [ReadOnlyZKClient-127.0.0.1:52059@0x0e6a4777] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-18 05:05:24,275 DEBUG [hconnection-0x7944646-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:24,278 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:24,279 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=35379] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:46636 deadline: 1689656784278, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:24,374 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-18 05:05:24,400 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-18 05:05:24,400 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 05:05:24,403 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta, suffix=.meta, logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034, archiveDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs, maxLogs=32 2023-07-18 05:05:24,421 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK] 2023-07-18 05:05:24,421 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK] 2023-07-18 05:05:24,424 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta 2023-07-18 05:05:24,424 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK], DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK]] 2023-07-18 05:05:24,425 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:24,425 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-18 05:05:24,425 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-18 05:05:24,425 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-18 05:05:24,426 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-18 05:05:24,426 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:24,426 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-18 05:05:24,426 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-18 05:05:24,428 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-18 05:05:24,429 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info 2023-07-18 05:05:24,429 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info 2023-07-18 05:05:24,429 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-18 05:05:24,431 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:24,431 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-18 05:05:24,432 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 05:05:24,432 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 05:05:24,432 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-18 05:05:24,433 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:24,433 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-18 05:05:24,435 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table 2023-07-18 05:05:24,435 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table 2023-07-18 05:05:24,435 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-18 05:05:24,436 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:24,437 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740 2023-07-18 05:05:24,440 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740 2023-07-18 05:05:24,442 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-18 05:05:24,450 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-18 05:05:24,451 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-18 05:05:24,511 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/.tmp/info/b975eb96fc244794ae4f4c08b9ee4542 2023-07-18 05:05:24,573 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/.tmp/table/edafa4a629f54327ae82a2ad50f54d7c 2023-07-18 05:05:24,582 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/.tmp/info/b975eb96fc244794ae4f4c08b9ee4542 as hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info/b975eb96fc244794ae4f4c08b9ee4542 2023-07-18 05:05:24,590 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info/b975eb96fc244794ae4f4c08b9ee4542, entries=20, sequenceid=13, filesize=6.9 K 2023-07-18 05:05:24,592 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/.tmp/table/edafa4a629f54327ae82a2ad50f54d7c as hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table/edafa4a629f54327ae82a2ad50f54d7c 2023-07-18 05:05:24,603 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table/edafa4a629f54327ae82a2ad50f54d7c, entries=4, sequenceid=13, filesize=4.7 K 2023-07-18 05:05:24,603 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 153ms, sequenceid=13, compaction requested=false; wal=null 2023-07-18 05:05:24,605 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-18 05:05:24,609 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-18 05:05:24,613 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-18 05:05:24,615 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-18 05:05:24,618 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-18 05:05:24,619 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63938320, jitterRate=-0.04724478721618652}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-18 05:05:24,619 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-18 05:05:24,621 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689656724394 2023-07-18 05:05:24,624 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-18 05:05:24,625 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-18 05:05:24,625 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35379,1689656720034, state=OPEN 2023-07-18 05:05:24,627 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 05:05:24,627 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 05:05:24,630 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-18 05:05:24,630 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35379,1689656720034 in 386 msec 2023-07-18 05:05:24,633 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-18 05:05:24,633 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 545 msec 2023-07-18 05:05:24,634 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,44031,1689656720079 had 3 regions 2023-07-18 05:05:24,635 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,44031,1689656720079, splitWal=true, meta=true, isMeta: false 2023-07-18 05:05:24,637 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,44031,1689656720079 WAL count=1, meta=false 2023-07-18 05:05:24,637 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540}] 2023-07-18 05:05:24,639 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:24,640 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540, worker=jenkins-hbase4.apache.org,35379,1689656720034}] 2023-07-18 05:05:24,793 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35379] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-18 05:05:24,808 DEBUG [hconnection-0x7944646-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:44031 this server is in the failed servers list 2023-07-18 05:05:24,812 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540, size=1.6 K (1665bytes) 2023-07-18 05:05:24,812 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540 2023-07-18 05:05:24,813 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540 after 1ms 2023-07-18 05:05:24,816 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540 took 4ms 2023-07-18 05:05:24,820 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 01bbc36252aecc1c98ab9ce391c47529: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-18 05:05:24,823 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 2bc2fc351e77525b9523e5bc259c0fe1: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-18 05:05:24,824 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540 so closing down 2023-07-18 05:05:24,824 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-18 05:05:24,827 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp 2023-07-18 05:05:24,827 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp 2023-07-18 05:05:24,828 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-18 05:05:24,851 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-18 05:05:24,859 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000005 2023-07-18 05:05:24,861 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-18 05:05:24,864 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000013 2023-07-18 05:05:24,864 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 47 ms; skipped=2; WAL=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540, size=1.6 K, length=1665, corrupted=false, cancelled=false 2023-07-18 05:05:24,865 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540, journal: Splitting hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540, size=1.6 K (1665bytes) at 1689656724812Finishing writing output for hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540 so closing down at 1689656724824 (+12 ms)Creating recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp at 1689656724827 (+3 ms)Creating recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp at 16896567248273 split writer threads finished at 1689656724829 (+2 ms)Closed recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689656724851 (+22 ms)Rename recovered edits hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000005 at 1689656724859 (+8 ms)Closed recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689656724861 (+2 ms)Rename recovered edits hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540.temp to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000013 at 1689656724864 (+3 ms)Processed 14 edits across 2 Regions in 47 ms; skipped=2; WAL=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540, size=1.6 K, length=1665, corrupted=false, cancelled=false at 1689656724864 2023-07-18 05:05:24,865 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540 2023-07-18 05:05:24,865 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-18 05:05:24,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.HMaster(3641): Remote procedure done, pid=18 2023-07-18 05:05:24,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] wal.WALSplitUtil(141): Moved hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting/jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540 to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs 2023-07-18 05:05:24,874 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-18 05:05:24,875 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540, worker=jenkins-hbase4.apache.org,35379,1689656720034 in 231 msec 2023-07-18 05:05:24,877 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:24,883 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-18 05:05:24,883 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C44031%2C1689656720079.1689656721540, worker=jenkins-hbase4.apache.org,35379,1689656720034 in 240 msec 2023-07-18 05:05:24,886 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,44031,1689656720079-splitting dir is empty, no logs to split. 2023-07-18 05:05:24,886 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,44031,1689656720079 WAL count=0, meta=false 2023-07-18 05:05:24,886 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,44031,1689656720079 WAL splitting is done? wals=0, meta=false 2023-07-18 05:05:24,889 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN}, {pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN}] 2023-07-18 05:05:24,891 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN 2023-07-18 05:05:24,892 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN 2023-07-18 05:05:24,893 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 05:05:24,894 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 05:05:25,016 DEBUG [hconnection-0x7944646-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:44031 this server is in the failed servers list 2023-07-18 05:05:25,043 DEBUG [jenkins-hbase4:43045] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-18 05:05:25,043 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=2bc2fc351e77525b9523e5bc259c0fe1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:25,043 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=01bbc36252aecc1c98ab9ce391c47529, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:25,044 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689656725043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689656725043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689656725043"}]},"ts":"1689656725043"} 2023-07-18 05:05:25,044 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689656725043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689656725043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689656725043"}]},"ts":"1689656725043"} 2023-07-18 05:05:25,045 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:44031 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:44031 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 05:05:25,047 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:44031 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:44031 2023-07-18 05:05:25,149 DEBUG [PEWorker-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:25,151 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:25,155 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 2bc2fc351e77525b9523e5bc259c0fe1, server=jenkins-hbase4.apache.org,35379,1689656720034}] 2023-07-18 05:05:25,156 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE; OpenRegionProcedure 01bbc36252aecc1c98ab9ce391c47529, server=jenkins-hbase4.apache.org,35379,1689656720034}] 2023-07-18 05:05:25,313 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:25,313 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 01bbc36252aecc1c98ab9ce391c47529, NAME => 'hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:25,314 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:25,314 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:25,314 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:25,314 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:25,316 INFO [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:25,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:25,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2bc2fc351e77525b9523e5bc259c0fe1, NAME => 't,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:25,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:25,318 DEBUG [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/info 2023-07-18 05:05:25,318 DEBUG [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/info 2023-07-18 05:05:25,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:25,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:25,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:25,319 INFO [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01bbc36252aecc1c98ab9ce391c47529 columnFamilyName info 2023-07-18 05:05:25,320 INFO [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] regionserver.HStore(310): Store=01bbc36252aecc1c98ab9ce391c47529/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:25,320 INFO [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:25,323 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:25,323 DEBUG [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/f 2023-07-18 05:05:25,323 DEBUG [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/f 2023-07-18 05:05:25,324 INFO [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2bc2fc351e77525b9523e5bc259c0fe1 columnFamilyName f 2023-07-18 05:05:25,325 DEBUG [hconnection-0x7944646-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:44031 this server is in the failed servers list 2023-07-18 05:05:25,325 INFO [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] regionserver.HStore(310): Store=2bc2fc351e77525b9523e5bc259c0fe1/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:25,328 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:25,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:25,330 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000005 2023-07-18 05:05:25,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:25,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000013 2023-07-18 05:05:25,335 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000005 2023-07-18 05:05:25,335 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 01bbc36252aecc1c98ab9ce391c47529 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-18 05:05:25,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000013 2023-07-18 05:05:25,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2bc2fc351e77525b9523e5bc259c0fe1 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-18 05:05:25,389 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/.tmp/info/dd2b6de13f594b16aca3f4aa74a1f320 2023-07-18 05:05:25,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/.tmp/f/23a5e277ec44459b96a3005d9744f2c3 2023-07-18 05:05:25,400 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/.tmp/info/dd2b6de13f594b16aca3f4aa74a1f320 as hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/info/dd2b6de13f594b16aca3f4aa74a1f320 2023-07-18 05:05:25,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/.tmp/f/23a5e277ec44459b96a3005d9744f2c3 as hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/f/23a5e277ec44459b96a3005d9744f2c3 2023-07-18 05:05:25,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/f/23a5e277ec44459b96a3005d9744f2c3, entries=10, sequenceid=13, filesize=4.8 K 2023-07-18 05:05:25,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 2bc2fc351e77525b9523e5bc259c0fe1 in 83ms, sequenceid=13, compaction requested=false; wal=null 2023-07-18 05:05:25,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-18 05:05:25,424 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/info/dd2b6de13f594b16aca3f4aa74a1f320, entries=2, sequenceid=5, filesize=4.8 K 2023-07-18 05:05:25,425 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 01bbc36252aecc1c98ab9ce391c47529 in 90ms, sequenceid=5, compaction requested=false; wal=null 2023-07-18 05:05:25,425 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-18 05:05:25,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/0000000000000000013 2023-07-18 05:05:25,428 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/0000000000000000005 2023-07-18 05:05:25,433 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:25,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:25,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-18 05:05:25,439 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-18 05:05:25,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2bc2fc351e77525b9523e5bc259c0fe1; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62623699, jitterRate=-0.06683416664600372}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 05:05:25,440 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 01bbc36252aecc1c98ab9ce391c47529; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63972405, jitterRate=-0.046736881136894226}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 05:05:25,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2bc2fc351e77525b9523e5bc259c0fe1: 2023-07-18 05:05:25,440 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 01bbc36252aecc1c98ab9ce391c47529: 2023-07-18 05:05:25,442 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1., pid=21, masterSystemTime=1689656725307 2023-07-18 05:05:25,442 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529., pid=22, masterSystemTime=1689656725307 2023-07-18 05:05:25,448 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=2bc2fc351e77525b9523e5bc259c0fe1, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:25,449 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689656725448"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689656725448"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689656725448"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689656725448"}]},"ts":"1689656725448"} 2023-07-18 05:05:25,450 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:25,450 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:25,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:25,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:25,451 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=01bbc36252aecc1c98ab9ce391c47529, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:25,451 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689656725451"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689656725451"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689656725451"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689656725451"}]},"ts":"1689656725451"} 2023-07-18 05:05:25,455 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-18 05:05:25,456 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 2bc2fc351e77525b9523e5bc259c0fe1, server=jenkins-hbase4.apache.org,35379,1689656720034 in 296 msec 2023-07-18 05:05:25,458 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-07-18 05:05:25,459 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN in 567 msec 2023-07-18 05:05:25,459 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; OpenRegionProcedure 01bbc36252aecc1c98ab9ce391c47529, server=jenkins-hbase4.apache.org,35379,1689656720034 in 298 msec 2023-07-18 05:05:25,462 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-07-18 05:05:25,462 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,44031,1689656720079 after splitting done 2023-07-18 05:05:25,462 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN in 570 msec 2023-07-18 05:05:25,462 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,44031,1689656720079 from processing; numProcessing=0 2023-07-18 05:05:25,465 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,44031,1689656720079, splitWal=true, meta=true in 1.6750 sec 2023-07-18 05:05:25,837 DEBUG [Listener at localhost/35655] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e6a4777 to 127.0.0.1:52059 2023-07-18 05:05:25,838 DEBUG [Listener at localhost/35655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:25,838 INFO [Listener at localhost/35655] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-18 05:05:25,856 INFO [Listener at localhost/35655] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=267 (was 291), OpenFileDescriptor=583 (was 581) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=152 (was 152), ProcessCount=170 (was 170), AvailableMemoryMB=6842 (was 6111) - AvailableMemoryMB LEAK? - 2023-07-18 05:05:25,868 INFO [Listener at localhost/35655] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=267, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=152, ProcessCount=170, AvailableMemoryMB=6837 2023-07-18 05:05:25,869 INFO [Listener at localhost/35655] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:44031 2023-07-18 05:05:25,887 INFO [Listener at localhost/35655] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-18 05:05:25,887 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:25,887 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:25,888 INFO [Listener at localhost/35655] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-18 05:05:25,888 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 05:05:25,888 INFO [Listener at localhost/35655] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-18 05:05:25,888 INFO [Listener at localhost/35655] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-18 05:05:25,890 INFO [Listener at localhost/35655] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45507 2023-07-18 05:05:25,890 INFO [Listener at localhost/35655] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-18 05:05:25,891 DEBUG [Listener at localhost/35655] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-18 05:05:25,892 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:25,894 INFO [Listener at localhost/35655] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 05:05:25,895 INFO [Listener at localhost/35655] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45507 connecting to ZooKeeper ensemble=127.0.0.1:52059 2023-07-18 05:05:25,899 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:455070x0, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-18 05:05:25,900 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(162): regionserver:455070x0, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-18 05:05:25,901 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(162): regionserver:455070x0, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-18 05:05:25,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:45507-0x10176d1f5910008 connected 2023-07-18 05:05:25,908 DEBUG [Listener at localhost/35655] zookeeper.ZKUtil(164): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-18 05:05:25,910 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45507 2023-07-18 05:05:25,911 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45507 2023-07-18 05:05:25,911 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45507 2023-07-18 05:05:25,914 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45507 2023-07-18 05:05:25,914 DEBUG [Listener at localhost/35655] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45507 2023-07-18 05:05:25,916 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(951): ClusterId : 840b74ac-9460-422b-b113-cc18493fd4dc 2023-07-18 05:05:25,917 DEBUG [RS:2;jenkins-hbase4:45507] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-18 05:05:25,920 DEBUG [RS:2;jenkins-hbase4:45507] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-18 05:05:25,920 DEBUG [RS:2;jenkins-hbase4:45507] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-18 05:05:25,922 DEBUG [RS:2;jenkins-hbase4:45507] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-18 05:05:25,923 DEBUG [RS:2;jenkins-hbase4:45507] zookeeper.ReadOnlyZKClient(139): Connect 0x0c4f7e07 to 127.0.0.1:52059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 05:05:25,928 DEBUG [RS:2;jenkins-hbase4:45507] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28891fe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 05:05:25,928 DEBUG [RS:2;jenkins-hbase4:45507] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b71577, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-18 05:05:25,937 DEBUG [RS:2;jenkins-hbase4:45507] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:45507 2023-07-18 05:05:25,937 INFO [RS:2;jenkins-hbase4:45507] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-18 05:05:25,937 INFO [RS:2;jenkins-hbase4:45507] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-18 05:05:25,937 DEBUG [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1022): About to register with Master. 2023-07-18 05:05:25,938 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,43045,1689656718865 with isa=jenkins-hbase4.apache.org/172.31.14.131:45507, startcode=1689656725886 2023-07-18 05:05:25,938 DEBUG [RS:2;jenkins-hbase4:45507] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-18 05:05:25,942 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33619, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-18 05:05:25,942 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43045] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:25,943 DEBUG [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9 2023-07-18 05:05:25,943 DEBUG [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:36887 2023-07-18 05:05:25,943 DEBUG [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-18 05:05:25,944 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:25,944 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:25,945 DEBUG [RS:2;jenkins-hbase4:45507] zookeeper.ZKUtil(162): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:25,945 WARN [RS:2;jenkins-hbase4:45507] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-18 05:05:25,945 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,45507,1689656725886] 2023-07-18 05:05:25,945 INFO [RS:2;jenkins-hbase4:45507] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 05:05:25,945 DEBUG [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1948): logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:25,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:25,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:25,952 DEBUG [RS:2;jenkins-hbase4:45507] zookeeper.ZKUtil(162): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:25,952 DEBUG [RS:2;jenkins-hbase4:45507] zookeeper.ZKUtil(162): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:25,953 DEBUG [RS:2;jenkins-hbase4:45507] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-18 05:05:25,954 INFO [RS:2;jenkins-hbase4:45507] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-18 05:05:25,963 INFO [RS:2;jenkins-hbase4:45507] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-18 05:05:25,967 INFO [RS:2;jenkins-hbase4:45507] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-18 05:05:25,967 INFO [RS:2;jenkins-hbase4:45507] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:25,970 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-18 05:05:25,973 INFO [RS:2;jenkins-hbase4:45507] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:25,973 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:25,973 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:25,973 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:25,973 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:25,973 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:25,973 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-18 05:05:25,974 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:25,974 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:25,974 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:25,974 DEBUG [RS:2;jenkins-hbase4:45507] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 05:05:25,983 INFO [RS:2;jenkins-hbase4:45507] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:25,984 INFO [RS:2;jenkins-hbase4:45507] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:25,984 INFO [RS:2;jenkins-hbase4:45507] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:25,997 INFO [RS:2;jenkins-hbase4:45507] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-18 05:05:25,998 INFO [RS:2;jenkins-hbase4:45507] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45507,1689656725886-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 05:05:26,015 INFO [RS:2;jenkins-hbase4:45507] regionserver.Replication(203): jenkins-hbase4.apache.org,45507,1689656725886 started 2023-07-18 05:05:26,015 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,45507,1689656725886, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:45507, sessionid=0x10176d1f5910008 2023-07-18 05:05:26,015 INFO [Listener at localhost/35655] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:45507,5,FailOnTimeoutGroup] 2023-07-18 05:05:26,015 DEBUG [RS:2;jenkins-hbase4:45507] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-18 05:05:26,015 INFO [Listener at localhost/35655] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:26,015 DEBUG [RS:2;jenkins-hbase4:45507] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:26,016 DEBUG [RS:2;jenkins-hbase4:45507] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45507,1689656725886' 2023-07-18 05:05:26,016 INFO [Listener at localhost/35655] client.TestScannerTimeout(177): START ************ test3686b 2023-07-18 05:05:26,016 DEBUG [RS:2;jenkins-hbase4:45507] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-18 05:05:26,017 DEBUG [RS:2;jenkins-hbase4:45507] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-18 05:05:26,017 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:44031 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:44031 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 05:05:26,017 DEBUG [RS:2;jenkins-hbase4:45507] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-18 05:05:26,018 DEBUG [RS:2;jenkins-hbase4:45507] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-18 05:05:26,018 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:44031 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:44031 2023-07-18 05:05:26,018 DEBUG [RS:2;jenkins-hbase4:45507] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:26,019 DEBUG [RS:2;jenkins-hbase4:45507] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45507,1689656725886' 2023-07-18 05:05:26,019 DEBUG [RS:2;jenkins-hbase4:45507] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-18 05:05:26,020 DEBUG [RS:2;jenkins-hbase4:45507] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-18 05:05:26,021 DEBUG [RS:2;jenkins-hbase4:45507] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-18 05:05:26,021 INFO [RS:2;jenkins-hbase4:45507] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-18 05:05:26,021 INFO [RS:2;jenkins-hbase4:45507] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-18 05:05:26,123 DEBUG [hconnection-0x32508046-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:26,125 INFO [RS:2;jenkins-hbase4:45507] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45507%2C1689656725886, suffix=, logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,45507,1689656725886, archiveDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs, maxLogs=32 2023-07-18 05:05:26,127 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:26,130 DEBUG [Listener at localhost/35655] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-18 05:05:26,130 DEBUG [Listener at localhost/35655] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:26,131 DEBUG [Listener at localhost/35655] zookeeper.ReadOnlyZKClient(139): Connect 0x1d5176de to 127.0.0.1:52059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 05:05:26,139 DEBUG [Listener at localhost/35655] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bbe7d10, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 05:05:26,143 DEBUG [hconnection-0x6ee5f3a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:26,147 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:26,161 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK] 2023-07-18 05:05:26,161 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK] 2023-07-18 05:05:26,168 INFO [RS:2;jenkins-hbase4:45507] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,45507,1689656725886/jenkins-hbase4.apache.org%2C45507%2C1689656725886.1689656726128 2023-07-18 05:05:26,173 DEBUG [RS:2;jenkins-hbase4:45507] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK], DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK]] 2023-07-18 05:05:27,270 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-18 05:05:29,018 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-18 05:05:29,018 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-18 05:05:41,165 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 8391772561968463880 expired clientIPAndPort=172.31.14.131:46648, userName=jenkins, regionInfo=t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:43,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35379] regionserver.RSRpcServices(3094): Client tried to access missing scanner 8391772561968463880 2023-07-18 05:05:43,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35379] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:46648 deadline: 1689656803167, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8391772561968463880'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-18 05:05:43,170 DEBUG [Listener at localhost/35655] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8391772561968463880'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8391772561968463880'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8391772561968463880'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-18 05:05:43,179 DEBUG [Listener at localhost/35655] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d5176de to 127.0.0.1:52059 2023-07-18 05:05:43,179 DEBUG [Listener at localhost/35655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:43,180 INFO [Listener at localhost/35655] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-18 05:05:43,195 INFO [Listener at localhost/35655] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=297 (was 267) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52059@0x0c4f7e07 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1933728359.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52059@0x1d5176de-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_430593769_17 at /127.0.0.1:47378 [Receiving block BP-780703847-172.31.14.131-1689656715810:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52059@0x0c4f7e07-SendThread(127.0.0.1:52059) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: PacketResponder: BP-780703847-172.31.14.131-1689656715810:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_430593769_17 at /127.0.0.1:50512 [Receiving block BP-780703847-172.31.14.131-1689656715810:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9-prefix:jenkins-hbase4.apache.org,45507,1689656725886 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-780703847-172.31.14.131-1689656715810:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase4:45507Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52059@0x1d5176de java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1933728359.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS:2;jenkins-hbase4:45507 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:45507-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52059@0x0c4f7e07-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:36887 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52059@0x1d5176de-SendThread(127.0.0.1:52059) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=45507 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=574 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=180 (was 152) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=6652 (was 6837) 2023-07-18 05:05:43,213 INFO [Listener at localhost/35655] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=297, OpenFileDescriptor=574, MaxFileDescriptor=60000, SystemLoadAverage=180, ProcessCount=170, AvailableMemoryMB=6636 2023-07-18 05:05:43,214 INFO [Listener at localhost/35655] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:44031 2023-07-18 05:05:43,214 INFO [Listener at localhost/35655] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,44031,1689656720079 2023-07-18 05:05:43,214 INFO [Listener at localhost/35655] client.TestScannerTimeout(104): START************ test2772 2023-07-18 05:05:43,220 DEBUG [Listener at localhost/35655] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-18 05:05:43,221 DEBUG [Listener at localhost/35655] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:43,221 DEBUG [Listener at localhost/35655] zookeeper.ReadOnlyZKClient(139): Connect 0x5587207f to 127.0.0.1:52059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 05:05:43,242 DEBUG [Listener at localhost/35655] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2924c472, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 05:05:43,243 ERROR [Listener at localhost/35655] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,35379,1689656720034: die! ***** 2023-07-18 05:05:43,243 ERROR [Listener at localhost/35655] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-18 05:05:43,243 DEBUG [Listener at localhost/35655] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-18 05:05:43,244 DEBUG [Listener at localhost/35655] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-18 05:05:43,250 DEBUG [Listener at localhost/35655] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-18 05:05:43,252 DEBUG [Listener at localhost/35655] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-18 05:05:43,280 INFO [Listener at localhost/35655] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 991952896, "init": 513802240, "max": 2051014656, "used": 300609296 }, "NonHeapMemoryUsage": { "committed": 124149760, "init": 2555904, "max": -1, "used": 121591856 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13537, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 4, "ProcessCallTime_num_ops": 65, "ProcessCallTime_min": 1, "ProcessCallTime_max": 3, "ProcessCallTime_mean": 2, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 3, "ProcessCallTime_90th_percentile": 3, "ProcessCallTime_95th_percentile": 3, "ProcessCallTime_98th_percentile": 3, "ProcessCallTime_99th_percentile": 3, "ProcessCallTime_99.9th_percentile": 3, "ProcessCallTime_TimeRangeCount_0-1": 1, "ProcessCallTime_TimeRangeCount_1-3": 1, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 65, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 65, "TotalCallTime_min": 1, "TotalCallTime_max": 3, "TotalCallTime_mean": 2, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 3, "TotalCallTime_90th_percentile": 3, "TotalCallTime_95th_percentile": 3, "TotalCallTime_98th_percentile": 3, "TotalCallTime_99th_percentile": 3, "TotalCallTime_99.9th_percentile": 3, "TotalCallTime_TimeRangeCount_0-1": 1, "TotalCallTime_TimeRangeCount_1-3": 1, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 65, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 6, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 65, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 18483 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:52059", "tag.serverName": "jenkins-hbase4.apache.org,44031,1689656720079", "tag.clusterId": "840b74ac-9460-422b-b113-cc18493fd4dc", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689656720079, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 2492, "localBytesRead": 2492, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 8750, "ByteBuffAllocatorPoolAllocationBytes": 998400, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 40, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 6, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 12, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 22, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 12, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 22, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 12, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.4642858746794323, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.1865078722505311, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-18 05:05:43,282 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43045] master.MasterRpcServices(612): jenkins-hbase4.apache.org,35379,1689656720034 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,35379,1689656720034: die! ***** 2023-07-18 05:05:43,283 INFO [Listener at localhost/35655] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,35379,1689656720034' ***** 2023-07-18 05:05:43,284 INFO [Listener at localhost/35655] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-18 05:05:43,284 INFO [RS:0;jenkins-hbase4:35379] regionserver.HeapMemoryManager(220): Stopping 2023-07-18 05:05:43,284 INFO [RS:0;jenkins-hbase4:35379] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-18 05:05:43,284 INFO [RS:0;jenkins-hbase4:35379] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-18 05:05:43,284 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(3305): Received CLOSE for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:43,284 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-18 05:05:43,286 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(3305): Received CLOSE for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:43,287 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:43,287 DEBUG [RS:0;jenkins-hbase4:35379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70298017 to 127.0.0.1:52059 2023-07-18 05:05:43,287 DEBUG [RS:0;jenkins-hbase4:35379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:43,287 INFO [RS:0;jenkins-hbase4:35379] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-18 05:05:43,287 INFO [RS:0;jenkins-hbase4:35379] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-18 05:05:43,287 INFO [RS:0;jenkins-hbase4:35379] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-18 05:05:43,287 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-18 05:05:43,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2bc2fc351e77525b9523e5bc259c0fe1, disabling compactions & flushes 2023-07-18 05:05:43,289 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:43,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:43,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. after waiting 0 ms 2023-07-18 05:05:43,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:43,289 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-18 05:05:43,290 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1478): Online Regions={2bc2fc351e77525b9523e5bc259c0fe1=t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1., 1588230740=hbase:meta,,1.1588230740, 01bbc36252aecc1c98ab9ce391c47529=hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.} 2023-07-18 05:05:43,290 DEBUG [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1504): Waiting on 01bbc36252aecc1c98ab9ce391c47529, 1588230740, 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:43,291 DEBUG [hconnection-0x72d7eac2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:43,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:43,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2bc2fc351e77525b9523e5bc259c0fe1: 2023-07-18 05:05:43,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:43,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 01bbc36252aecc1c98ab9ce391c47529, disabling compactions & flushes 2023-07-18 05:05:43,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:43,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:43,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. after waiting 0 ms 2023-07-18 05:05:43,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:43,304 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-18 05:05:43,304 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-18 05:05:43,304 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-18 05:05:43,304 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-18 05:05:43,304 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-18 05:05:43,311 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47132, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:43,312 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=35379] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35379,1689656720034 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 05:05:43,312 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=35379] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:47132 deadline: 1689656803312, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35379,1689656720034 aborting 2023-07-18 05:05:43,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:43,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 01bbc36252aecc1c98ab9ce391c47529: 2023-07-18 05:05:43,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:43,321 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 1792 in region hbase:meta,,1.1588230740 2023-07-18 05:05:43,321 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-18 05:05:43,322 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-18 05:05:43,322 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-18 05:05:43,322 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-18 05:05:43,375 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-18 05:05:43,376 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-18 05:05:43,418 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35379] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35379,1689656720034 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 05:05:43,418 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35379] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:47132 deadline: 1689656803418, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35379,1689656720034 aborting 2023-07-18 05:05:43,490 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,35379,1689656720034; all regions closed. 2023-07-18 05:05:43,511 DEBUG [RS:0;jenkins-hbase4:35379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:43,512 INFO [RS:0;jenkins-hbase4:35379] regionserver.LeaseManager(133): Closed leases 2023-07-18 05:05:43,512 INFO [RS:0;jenkins-hbase4:35379] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-18 05:05:43,513 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-18 05:05:43,513 INFO [RS:0;jenkins-hbase4:35379] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35379 2023-07-18 05:05:43,517 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:43,517 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 2023-07-18 05:05:43,517 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:43,517 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:43,517 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:43,519 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35379,1689656720034] 2023-07-18 05:05:43,519 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35379,1689656720034; numProcessing=1 2023-07-18 05:05:43,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:43,520 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,35379,1689656720034 znode expired, triggering replicatorRemoved event 2023-07-18 05:05:43,522 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35379,1689656720034 already deleted, retry=false 2023-07-18 05:05:43,522 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,35379,1689656720034 on jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:43,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:43,524 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=23, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,35379,1689656720034, splitWal=true, meta=true 2023-07-18 05:05:43,524 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=23 for jenkins-hbase4.apache.org,35379,1689656720034 (carryingMeta=true) jenkins-hbase4.apache.org,35379,1689656720034/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@78b675dc[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-18 05:05:43,526 INFO [PEWorker-3] procedure.ServerCrashProcedure(161): Start pid=23, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35379,1689656720034, splitWal=true, meta=true 2023-07-18 05:05:43,527 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35379,1689656720034, splitWal=true, meta=true, isMeta: true 2023-07-18 05:05:43,534 DEBUG [PEWorker-3] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting 2023-07-18 05:05:43,536 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,35379,1689656720034 WAL count=1, meta=true 2023-07-18 05:05:43,536 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta}] 2023-07-18 05:05:43,536 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-18 05:05:43,538 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:43,540 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta, worker=jenkins-hbase4.apache.org,45507,1689656725886}] 2023-07-18 05:05:43,621 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35379 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35379 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 05:05:43,623 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35379 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35379 2023-07-18 05:05:43,623 INFO [RS:0;jenkins-hbase4:35379] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,35379,1689656720034; zookeeper connection closed. 2023-07-18 05:05:43,623 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 05:05:43,623 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:35379-0x10176d1f5910001, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 05:05:43,623 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6d00e607] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6d00e607 2023-07-18 05:05:43,693 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:43,693 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-18 05:05:43,695 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44780, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-18 05:05:43,695 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45507] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-18 05:05:43,717 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta, size=2.3 K (2403bytes) 2023-07-18 05:05:43,717 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta 2023-07-18 05:05:43,718 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta after 1ms 2023-07-18 05:05:43,724 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta took 7ms 2023-07-18 05:05:43,729 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-18 05:05:43,731 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta so closing down 2023-07-18 05:05:43,732 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-18 05:05:43,743 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta.temp 2023-07-18 05:05:43,744 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-18 05:05:43,778 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) 2023-07-18 05:05:43,780 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta.temp to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-18 05:05:43,781 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 5 edits across 1 Regions in 56 ms; skipped=1; WAL=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false 2023-07-18 05:05:43,781 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta, journal: Splitting hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta, size=2.3 K (2403bytes) at 1689656743717Finishing writing output for hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta so closing down at 1689656743731 (+14 ms)Creating recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta.temp at 1689656743743 (+12 ms)3 split writer threads finished at 1689656743744 (+1 ms)Closed recovered edits writer path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) at 1689656743779 (+35 ms)Rename recovered edits hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta.temp to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 at 1689656743781 (+2 ms)Processed 5 edits across 1 Regions in 56 ms; skipped=1; WAL=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false at 1689656743781 2023-07-18 05:05:43,781 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta 2023-07-18 05:05:43,781 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-18 05:05:43,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.HMaster(3641): Remote procedure done, pid=25 2023-07-18 05:05:43,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] wal.WALSplitUtil(141): Moved hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs 2023-07-18 05:05:43,790 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-18 05:05:43,790 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta, worker=jenkins-hbase4.apache.org,45507,1689656725886 in 248 msec 2023-07-18 05:05:43,793 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:43,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=23 2023-07-18 05:05:43,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C35379%2C1689656720034.meta.1689656724404.meta, worker=jenkins-hbase4.apache.org,45507,1689656725886 in 257 msec 2023-07-18 05:05:43,802 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting dir is empty, no logs to split. 2023-07-18 05:05:43,802 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,35379,1689656720034 WAL count=0, meta=true 2023-07-18 05:05:43,802 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,35379,1689656720034 WAL splitting is done? wals=0, meta=true 2023-07-18 05:05:43,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-18 05:05:43,805 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-18 05:05:43,806 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 05:05:43,926 DEBUG [hconnection-0x72d7eac2-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35379 this server is in the failed servers list 2023-07-18 05:05:43,956 DEBUG [jenkins-hbase4:43045] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-18 05:05:43,957 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45507,1689656725886, state=OPENING 2023-07-18 05:05:43,961 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 05:05:43,961 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45507,1689656725886}] 2023-07-18 05:05:43,961 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 05:05:44,120 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-18 05:05:44,120 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 05:05:44,122 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45507%2C1689656725886.meta, suffix=.meta, logDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,45507,1689656725886, archiveDir=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs, maxLogs=32 2023-07-18 05:05:44,145 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK] 2023-07-18 05:05:44,146 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK] 2023-07-18 05:05:44,149 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,45507,1689656725886/jenkins-hbase4.apache.org%2C45507%2C1689656725886.meta.1689656744123.meta 2023-07-18 05:05:44,150 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35281,DS-ebdde9ea-77a3-4732-9998-b670c58d283a,DISK], DatanodeInfoWithStorage[127.0.0.1:39801,DS-5b5eba8d-4b2e-47a7-a625-02920d1a9333,DISK]] 2023-07-18 05:05:44,151 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:44,151 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-18 05:05:44,151 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-18 05:05:44,151 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-18 05:05:44,151 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-18 05:05:44,151 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:44,152 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-18 05:05:44,152 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-18 05:05:44,157 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-18 05:05:44,159 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info 2023-07-18 05:05:44,159 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info 2023-07-18 05:05:44,160 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-18 05:05:44,172 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info/b975eb96fc244794ae4f4c08b9ee4542 2023-07-18 05:05:44,172 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:44,172 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-18 05:05:44,174 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 05:05:44,175 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 05:05:44,176 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-18 05:05:44,177 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:44,177 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-18 05:05:44,179 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table 2023-07-18 05:05:44,179 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table 2023-07-18 05:05:44,179 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-18 05:05:44,202 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/table/edafa4a629f54327ae82a2ad50f54d7c 2023-07-18 05:05:44,202 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:44,204 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740 2023-07-18 05:05:44,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740 2023-07-18 05:05:44,213 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-18 05:05:44,218 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 16, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=19, path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-18 05:05:44,218 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-18 05:05:44,237 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=19 (bloomFilter=false), to=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/.tmp/info/b3a04037299544dbac2309c2f6298bd0 2023-07-18 05:05:44,245 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/.tmp/info/b3a04037299544dbac2309c2f6298bd0 as hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info/b3a04037299544dbac2309c2f6298bd0 2023-07-18 05:05:44,252 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info/b3a04037299544dbac2309c2f6298bd0, entries=16, sequenceid=19, filesize=6.4 K 2023-07-18 05:05:44,253 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 35ms, sequenceid=19, compaction requested=false; wal=null 2023-07-18 05:05:44,254 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-18 05:05:44,259 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-18 05:05:44,261 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-18 05:05:44,263 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=13 2023-07-18 05:05:44,264 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=20; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67566757, jitterRate=0.006823137402534485}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-18 05:05:44,264 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-18 05:05:44,265 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=27, masterSystemTime=1689656744114 2023-07-18 05:05:44,267 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-18 05:05:44,268 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-18 05:05:44,268 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45507,1689656725886, state=OPEN 2023-07-18 05:05:44,270 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 05:05:44,270 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 05:05:44,272 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-18 05:05:44,272 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45507,1689656725886 in 309 msec 2023-07-18 05:05:44,274 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=23 2023-07-18 05:05:44,274 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,35379,1689656720034 had 3 regions 2023-07-18 05:05:44,274 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 469 msec 2023-07-18 05:05:44,275 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35379,1689656720034, splitWal=true, meta=true, isMeta: false 2023-07-18 05:05:44,278 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,35379,1689656720034 WAL count=1, meta=false 2023-07-18 05:05:44,278 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540}] 2023-07-18 05:05:44,280 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:44,281 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540, worker=jenkins-hbase4.apache.org,45507,1689656725886}] 2023-07-18 05:05:44,431 DEBUG [hconnection-0x72d7eac2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:44,434 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45507] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=29 2023-07-18 05:05:44,434 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:44,442 DEBUG [hconnection-0x72d7eac2-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35379 this server is in the failed servers list 2023-07-18 05:05:44,451 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540, size=721 (721bytes) 2023-07-18 05:05:44,452 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540 2023-07-18 05:05:44,456 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540 after 4ms 2023-07-18 05:05:44,460 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540 took 9ms 2023-07-18 05:05:44,467 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 2bc2fc351e77525b9523e5bc259c0fe1: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-18 05:05:44,468 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 01bbc36252aecc1c98ab9ce391c47529: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-18 05:05:44,468 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540 so closing down 2023-07-18 05:05:44,469 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-18 05:05:44,469 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-18 05:05:44,469 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 6 ms; skipped=2; WAL=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540, size=721, length=721, corrupted=false, cancelled=false 2023-07-18 05:05:44,469 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540, journal: Splitting hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540, size=721 (721bytes) at 1689656744451Finishing writing output for hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540 so closing down at 1689656744469 (+18 ms)3 split writer threads finished at 1689656744469Processed 2 edits across 0 Regions in 6 ms; skipped=2; WAL=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540, size=721, length=721, corrupted=false, cancelled=false at 1689656744469 2023-07-18 05:05:44,469 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540 2023-07-18 05:05:44,469 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=29 2023-07-18 05:05:44,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] master.HMaster(3641): Remote procedure done, pid=29 2023-07-18 05:05:44,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43045] wal.WALSplitUtil(141): Moved hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting/jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540 to hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs 2023-07-18 05:05:44,477 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-07-18 05:05:44,477 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540, worker=jenkins-hbase4.apache.org,45507,1689656725886 in 193 msec 2023-07-18 05:05:44,478 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:44,481 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=23 2023-07-18 05:05:44,481 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C35379%2C1689656720034.1689656721540, worker=jenkins-hbase4.apache.org,45507,1689656725886 in 200 msec 2023-07-18 05:05:44,483 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/WALs/jenkins-hbase4.apache.org,35379,1689656720034-splitting dir is empty, no logs to split. 2023-07-18 05:05:44,483 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,35379,1689656720034 WAL count=0, meta=false 2023-07-18 05:05:44,483 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,35379,1689656720034 WAL splitting is done? wals=0, meta=false 2023-07-18 05:05:44,485 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN}, {pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN}] 2023-07-18 05:05:44,486 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN 2023-07-18 05:05:44,486 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN 2023-07-18 05:05:44,487 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 05:05:44,487 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 05:05:44,549 DEBUG [hconnection-0x72d7eac2-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35379 this server is in the failed servers list 2023-07-18 05:05:44,637 DEBUG [jenkins-hbase4:43045] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-18 05:05:44,637 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=2bc2fc351e77525b9523e5bc259c0fe1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:44,637 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=01bbc36252aecc1c98ab9ce391c47529, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:44,638 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689656744637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689656744637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689656744637"}]},"ts":"1689656744637"} 2023-07-18 05:05:44,638 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689656744637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689656744637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689656744637"}]},"ts":"1689656744637"} 2023-07-18 05:05:44,639 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35379 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35379 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 05:05:44,639 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35379 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35379 2023-07-18 05:05:44,741 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 05:05:44,742 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 05:05:44,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE; OpenRegionProcedure 01bbc36252aecc1c98ab9ce391c47529, server=jenkins-hbase4.apache.org,45507,1689656725886}] 2023-07-18 05:05:44,746 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=30, state=RUNNABLE; OpenRegionProcedure 2bc2fc351e77525b9523e5bc259c0fe1, server=jenkins-hbase4.apache.org,45507,1689656725886}] 2023-07-18 05:05:44,755 DEBUG [hconnection-0x72d7eac2-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35379 this server is in the failed servers list 2023-07-18 05:05:44,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:44,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2bc2fc351e77525b9523e5bc259c0fe1, NAME => 't,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:44,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:44,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:44,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:44,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:44,904 INFO [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:44,905 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:44,905 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 01bbc36252aecc1c98ab9ce391c47529, NAME => 'hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.', STARTKEY => '', ENDKEY => ''} 2023-07-18 05:05:44,906 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:44,906 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 05:05:44,906 DEBUG [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/f 2023-07-18 05:05:44,906 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:44,906 DEBUG [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/f 2023-07-18 05:05:44,906 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:44,906 INFO [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2bc2fc351e77525b9523e5bc259c0fe1 columnFamilyName f 2023-07-18 05:05:44,907 INFO [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:44,909 DEBUG [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/info 2023-07-18 05:05:44,909 DEBUG [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/info 2023-07-18 05:05:44,909 INFO [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01bbc36252aecc1c98ab9ce391c47529 columnFamilyName info 2023-07-18 05:05:44,914 DEBUG [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] regionserver.HStore(539): loaded hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/f/23a5e277ec44459b96a3005d9744f2c3 2023-07-18 05:05:44,915 INFO [StoreOpener-2bc2fc351e77525b9523e5bc259c0fe1-1] regionserver.HStore(310): Store=2bc2fc351e77525b9523e5bc259c0fe1/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:44,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:44,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:44,918 DEBUG [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] regionserver.HStore(539): loaded hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/info/dd2b6de13f594b16aca3f4aa74a1f320 2023-07-18 05:05:44,918 INFO [StoreOpener-01bbc36252aecc1c98ab9ce391c47529-1] regionserver.HStore(310): Store=01bbc36252aecc1c98ab9ce391c47529/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 05:05:44,919 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:44,920 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:44,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:44,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-18 05:05:44,925 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:44,926 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2bc2fc351e77525b9523e5bc259c0fe1; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65901949, jitterRate=-0.017984434962272644}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 05:05:44,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2bc2fc351e77525b9523e5bc259c0fe1: 2023-07-18 05:05:44,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1., pid=33, masterSystemTime=1689656744897 2023-07-18 05:05:44,928 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-18 05:05:44,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:44,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:44,930 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 01bbc36252aecc1c98ab9ce391c47529; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74711231, jitterRate=0.11328409612178802}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 05:05:44,930 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 01bbc36252aecc1c98ab9ce391c47529: 2023-07-18 05:05:44,930 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=2bc2fc351e77525b9523e5bc259c0fe1, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:44,930 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689656744930"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689656744930"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689656744930"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689656744930"}]},"ts":"1689656744930"} 2023-07-18 05:05:44,931 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529., pid=32, masterSystemTime=1689656744897 2023-07-18 05:05:44,932 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:44,932 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:44,933 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=01bbc36252aecc1c98ab9ce391c47529, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:44,933 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689656744933"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689656744933"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689656744933"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689656744933"}]},"ts":"1689656744933"} 2023-07-18 05:05:44,935 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=30 2023-07-18 05:05:44,935 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=30, state=SUCCESS; OpenRegionProcedure 2bc2fc351e77525b9523e5bc259c0fe1, server=jenkins-hbase4.apache.org,45507,1689656725886 in 186 msec 2023-07-18 05:05:44,936 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=t, region=2bc2fc351e77525b9523e5bc259c0fe1, ASSIGN in 450 msec 2023-07-18 05:05:44,938 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-07-18 05:05:44,938 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; OpenRegionProcedure 01bbc36252aecc1c98ab9ce391c47529, server=jenkins-hbase4.apache.org,45507,1689656725886 in 190 msec 2023-07-18 05:05:44,940 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=23 2023-07-18 05:05:44,940 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,35379,1689656720034 after splitting done 2023-07-18 05:05:44,940 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=01bbc36252aecc1c98ab9ce391c47529, ASSIGN in 453 msec 2023-07-18 05:05:44,940 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,35379,1689656720034 from processing; numProcessing=0 2023-07-18 05:05:44,942 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,35379,1689656720034, splitWal=true, meta=true in 1.4180 sec 2023-07-18 05:05:45,068 DEBUG [Listener at localhost/35655] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5587207f to 127.0.0.1:52059 2023-07-18 05:05:45,069 DEBUG [Listener at localhost/35655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:45,069 INFO [Listener at localhost/35655] client.TestScannerTimeout(124): END ************ test2772 2023-07-18 05:05:45,082 INFO [Listener at localhost/35655] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=279 (was 297), OpenFileDescriptor=582 (was 574) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=180 (was 180), ProcessCount=170 (was 170), AvailableMemoryMB=6532 (was 6636) 2023-07-18 05:05:45,082 INFO [Listener at localhost/35655] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-18 05:05:45,082 INFO [Listener at localhost/35655] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-18 05:05:45,083 DEBUG [Listener at localhost/35655] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x044a9743 to 127.0.0.1:52059 2023-07-18 05:05:45,083 DEBUG [Listener at localhost/35655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:45,083 DEBUG [Listener at localhost/35655] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-18 05:05:45,083 DEBUG [Listener at localhost/35655] util.JVMClusterUtil(257): Found active master hash=335720420, stopped=false 2023-07-18 05:05:45,083 INFO [Listener at localhost/35655] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:45,086 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-18 05:05:45,086 INFO [Listener at localhost/35655] procedure2.ProcedureExecutor(629): Stopping 2023-07-18 05:05:45,086 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:45,086 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-18 05:05:45,087 DEBUG [Listener at localhost/35655] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23337beb to 127.0.0.1:52059 2023-07-18 05:05:45,087 DEBUG [Listener at localhost/35655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:45,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 05:05:45,087 INFO [Listener at localhost/35655] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,45507,1689656725886' ***** 2023-07-18 05:05:45,087 INFO [Listener at localhost/35655] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-18 05:05:45,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 05:05:45,087 INFO [RS:2;jenkins-hbase4:45507] regionserver.HeapMemoryManager(220): Stopping 2023-07-18 05:05:45,087 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-18 05:05:45,087 INFO [RS:2;jenkins-hbase4:45507] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-18 05:05:45,087 INFO [RS:2;jenkins-hbase4:45507] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-18 05:05:45,088 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(3305): Received CLOSE for 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:45,088 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(3305): Received CLOSE for 01bbc36252aecc1c98ab9ce391c47529 2023-07-18 05:05:45,088 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:45,088 DEBUG [RS:2;jenkins-hbase4:45507] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c4f7e07 to 127.0.0.1:52059 2023-07-18 05:05:45,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2bc2fc351e77525b9523e5bc259c0fe1, disabling compactions & flushes 2023-07-18 05:05:45,088 DEBUG [RS:2;jenkins-hbase4:45507] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:45,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:45,088 INFO [RS:2;jenkins-hbase4:45507] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-18 05:05:45,089 INFO [RS:2;jenkins-hbase4:45507] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-18 05:05:45,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:45,089 INFO [RS:2;jenkins-hbase4:45507] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-18 05:05:45,089 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. after waiting 0 ms 2023-07-18 05:05:45,089 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-18 05:05:45,089 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:45,089 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-18 05:05:45,089 DEBUG [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1478): Online Regions={2bc2fc351e77525b9523e5bc259c0fe1=t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1., 1588230740=hbase:meta,,1.1588230740, 01bbc36252aecc1c98ab9ce391c47529=hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529.} 2023-07-18 05:05:45,089 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-18 05:05:45,090 DEBUG [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1504): Waiting on 01bbc36252aecc1c98ab9ce391c47529, 1588230740, 2bc2fc351e77525b9523e5bc259c0fe1 2023-07-18 05:05:45,090 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-18 05:05:45,090 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-18 05:05:45,090 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-18 05:05:45,090 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-18 05:05:45,090 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-18 05:05:45,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/default/t/2bc2fc351e77525b9523e5bc259c0fe1/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-18 05:05:45,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:45,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2bc2fc351e77525b9523e5bc259c0fe1: 2023-07-18 05:05:45,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689656722807.2bc2fc351e77525b9523e5bc259c0fe1. 2023-07-18 05:05:45,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 01bbc36252aecc1c98ab9ce391c47529, disabling compactions & flushes 2023-07-18 05:05:45,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:45,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:45,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. after waiting 0 ms 2023-07-18 05:05:45,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:45,102 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/namespace/01bbc36252aecc1c98ab9ce391c47529/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-18 05:05:45,103 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:45,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 01bbc36252aecc1c98ab9ce391c47529: 2023-07-18 05:05:45,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689656721957.01bbc36252aecc1c98ab9ce391c47529. 2023-07-18 05:05:45,120 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=26 (bloomFilter=false), to=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/.tmp/info/0ff592430ec04fbe953d6ffc6a615a8c 2023-07-18 05:05:45,128 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/.tmp/info/0ff592430ec04fbe953d6ffc6a615a8c as hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info/0ff592430ec04fbe953d6ffc6a615a8c 2023-07-18 05:05:45,134 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/info/0ff592430ec04fbe953d6ffc6a615a8c, entries=16, sequenceid=26, filesize=6.4 K 2023-07-18 05:05:45,136 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 46ms, sequenceid=26, compaction requested=true 2023-07-18 05:05:45,145 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/data/hbase/meta/1588230740/recovered.edits/29.seqid, newMaxSeqId=29, maxSeqId=19 2023-07-18 05:05:45,146 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-18 05:05:45,146 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-18 05:05:45,146 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-18 05:05:45,146 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-18 05:05:45,290 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,45507,1689656725886; all regions closed. 2023-07-18 05:05:45,297 DEBUG [RS:2;jenkins-hbase4:45507] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs 2023-07-18 05:05:45,297 INFO [RS:2;jenkins-hbase4:45507] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C45507%2C1689656725886.meta:.meta(num 1689656744123) 2023-07-18 05:05:45,302 DEBUG [RS:2;jenkins-hbase4:45507] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/oldWALs 2023-07-18 05:05:45,302 INFO [RS:2;jenkins-hbase4:45507] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C45507%2C1689656725886:(num 1689656726128) 2023-07-18 05:05:45,302 DEBUG [RS:2;jenkins-hbase4:45507] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:45,302 INFO [RS:2;jenkins-hbase4:45507] regionserver.LeaseManager(133): Closed leases 2023-07-18 05:05:45,303 INFO [RS:2;jenkins-hbase4:45507] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-18 05:05:45,303 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-18 05:05:45,303 INFO [RS:2;jenkins-hbase4:45507] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45507 2023-07-18 05:05:45,306 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:45,306 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45507,1689656725886 2023-07-18 05:05:45,306 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 05:05:45,308 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,45507,1689656725886] 2023-07-18 05:05:45,308 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,45507,1689656725886; numProcessing=1 2023-07-18 05:05:45,309 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,45507,1689656725886 already deleted, retry=false 2023-07-18 05:05:45,309 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,45507,1689656725886 expired; onlineServers=0 2023-07-18 05:05:45,309 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,43045,1689656718865' ***** 2023-07-18 05:05:45,309 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-18 05:05:45,310 DEBUG [M:0;jenkins-hbase4:43045] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@553fa5df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-18 05:05:45,310 INFO [M:0;jenkins-hbase4:43045] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,43045,1689656718865 2023-07-18 05:05:45,310 INFO [M:0;jenkins-hbase4:43045] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,43045,1689656718865; all regions closed. 2023-07-18 05:05:45,310 DEBUG [M:0;jenkins-hbase4:43045] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 05:05:45,310 DEBUG [M:0;jenkins-hbase4:43045] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-18 05:05:45,311 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-18 05:05:45,311 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689656721172] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689656721172,5,FailOnTimeoutGroup] 2023-07-18 05:05:45,311 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689656721171] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689656721171,5,FailOnTimeoutGroup] 2023-07-18 05:05:45,311 DEBUG [M:0;jenkins-hbase4:43045] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-18 05:05:45,311 INFO [M:0;jenkins-hbase4:43045] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-18 05:05:45,311 INFO [M:0;jenkins-hbase4:43045] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-18 05:05:45,311 INFO [M:0;jenkins-hbase4:43045] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-18 05:05:45,311 DEBUG [M:0;jenkins-hbase4:43045] master.HMaster(1512): Stopping service threads 2023-07-18 05:05:45,311 INFO [M:0;jenkins-hbase4:43045] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-18 05:05:45,312 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-18 05:05:45,312 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 05:05:45,312 ERROR [M:0;jenkins-hbase4:43045] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1168075184) connection to localhost/127.0.0.1:36887 from jenkins,5,PEWorkerGroup] 2023-07-18 05:05:45,312 INFO [M:0;jenkins-hbase4:43045] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-18 05:05:45,312 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-18 05:05:45,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-18 05:05:45,313 DEBUG [M:0;jenkins-hbase4:43045] zookeeper.ZKUtil(398): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-18 05:05:45,313 WARN [M:0;jenkins-hbase4:43045] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-18 05:05:45,313 INFO [M:0;jenkins-hbase4:43045] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-18 05:05:45,313 INFO [M:0;jenkins-hbase4:43045] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-18 05:05:45,314 DEBUG [M:0;jenkins-hbase4:43045] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-18 05:05:45,314 INFO [M:0;jenkins-hbase4:43045] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 05:05:45,314 DEBUG [M:0;jenkins-hbase4:43045] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 05:05:45,314 DEBUG [M:0;jenkins-hbase4:43045] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-18 05:05:45,314 DEBUG [M:0;jenkins-hbase4:43045] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 05:05:45,314 INFO [M:0;jenkins-hbase4:43045] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=111.29 KB heapSize=134.22 KB 2023-07-18 05:05:45,332 INFO [M:0;jenkins-hbase4:43045] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=111.29 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ebd199637ea84f3b928e1470435a992e 2023-07-18 05:05:45,339 DEBUG [M:0;jenkins-hbase4:43045] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ebd199637ea84f3b928e1470435a992e as hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ebd199637ea84f3b928e1470435a992e 2023-07-18 05:05:45,345 INFO [M:0;jenkins-hbase4:43045] regionserver.HStore(1080): Added hdfs://localhost:36887/user/jenkins/test-data/ac5ca4a9-8b47-3892-5874-5eaa9d0b6fb9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ebd199637ea84f3b928e1470435a992e, entries=33, sequenceid=257, filesize=8.5 K 2023-07-18 05:05:45,346 INFO [M:0;jenkins-hbase4:43045] regionserver.HRegion(2948): Finished flush of dataSize ~111.29 KB/113959, heapSize ~134.20 KB/137424, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 32ms, sequenceid=257, compaction requested=false 2023-07-18 05:05:45,348 INFO [M:0;jenkins-hbase4:43045] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 05:05:45,348 DEBUG [M:0;jenkins-hbase4:43045] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-18 05:05:45,351 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-18 05:05:45,351 INFO [M:0;jenkins-hbase4:43045] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-18 05:05:45,352 INFO [M:0;jenkins-hbase4:43045] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43045 2023-07-18 05:05:45,354 DEBUG [M:0;jenkins-hbase4:43045] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,43045,1689656718865 already deleted, retry=false 2023-07-18 05:05:45,573 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 05:05:45,573 INFO [M:0;jenkins-hbase4:43045] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,43045,1689656718865; zookeeper connection closed. 2023-07-18 05:05:45,573 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): master:43045-0x10176d1f5910000, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 05:05:45,674 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 05:05:45,674 INFO [RS:2;jenkins-hbase4:45507] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,45507,1689656725886; zookeeper connection closed. 2023-07-18 05:05:45,674 DEBUG [Listener at localhost/35655-EventThread] zookeeper.ZKWatcher(600): regionserver:45507-0x10176d1f5910008, quorum=127.0.0.1:52059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 05:05:45,674 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4592f6c5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4592f6c5 2023-07-18 05:05:45,675 INFO [Listener at localhost/35655] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-18 05:05:45,675 WARN [Listener at localhost/35655] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-18 05:05:45,678 INFO [Listener at localhost/35655] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-18 05:05:45,783 WARN [BP-780703847-172.31.14.131-1689656715810 heartbeating to localhost/127.0.0.1:36887] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-18 05:05:45,783 WARN [BP-780703847-172.31.14.131-1689656715810 heartbeating to localhost/127.0.0.1:36887] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-780703847-172.31.14.131-1689656715810 (Datanode Uuid 3b0f0fec-8b12-4bb0-9296-5c650e0a5410) service to localhost/127.0.0.1:36887 2023-07-18 05:05:45,785 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/dfs/data/data3/current/BP-780703847-172.31.14.131-1689656715810] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-18 05:05:45,785 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/dfs/data/data4/current/BP-780703847-172.31.14.131-1689656715810] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-18 05:05:45,787 WARN [Listener at localhost/35655] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-18 05:05:45,791 INFO [Listener at localhost/35655] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-18 05:05:45,896 WARN [BP-780703847-172.31.14.131-1689656715810 heartbeating to localhost/127.0.0.1:36887] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-18 05:05:45,896 WARN [BP-780703847-172.31.14.131-1689656715810 heartbeating to localhost/127.0.0.1:36887] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-780703847-172.31.14.131-1689656715810 (Datanode Uuid d4e5ff0e-b81a-4ef5-83b6-3e12d0570039) service to localhost/127.0.0.1:36887 2023-07-18 05:05:45,897 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/dfs/data/data1/current/BP-780703847-172.31.14.131-1689656715810] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-18 05:05:45,898 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90fe3e96-7d28-6800-e43f-837029266767/cluster_7b8bb8fe-4f33-3aa7-e9c7-dedb06c09a80/dfs/data/data2/current/BP-780703847-172.31.14.131-1689656715810] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-18 05:05:45,937 INFO [Listener at localhost/35655] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-18 05:05:45,986 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-18 05:05:46,050 INFO [Listener at localhost/35655] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-18 05:05:46,091 INFO [Listener at localhost/35655] hbase.HBaseTestingUtility(1293): Minicluster is down