2023-07-20 17:05:12,898 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-20 17:05:13,132 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67 2023-07-20 17:05:13,144 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-20 17:05:13,145 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56, deleteOnExit=true 2023-07-20 17:05:13,145 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-20 17:05:13,145 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/test.cache.data in system properties and HBase conf 2023-07-20 17:05:13,146 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/hadoop.tmp.dir in system properties and HBase conf 2023-07-20 17:05:13,146 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/hadoop.log.dir in system properties and HBase conf 2023-07-20 17:05:13,147 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-20 17:05:13,147 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-20 17:05:13,148 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-20 17:05:13,257 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-20 17:05:13,651 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-20 17:05:13,656 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-20 17:05:13,656 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-20 17:05:13,656 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-20 17:05:13,656 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-20 17:05:13,657 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-20 17:05:13,657 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-20 17:05:13,657 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-20 17:05:13,657 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-20 17:05:13,658 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-20 17:05:13,658 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/nfs.dump.dir in system properties and HBase conf 2023-07-20 17:05:13,659 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/java.io.tmpdir in system properties and HBase conf 2023-07-20 17:05:13,659 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-20 17:05:13,660 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-20 17:05:13,660 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-20 17:05:14,144 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-20 17:05:14,149 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-20 17:05:14,437 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-20 17:05:14,587 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-20 17:05:14,601 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 17:05:14,635 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 17:05:14,666 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/java.io.tmpdir/Jetty_localhost_33361_hdfs____.bpdbig/webapp 2023-07-20 17:05:14,816 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33361 2023-07-20 17:05:14,826 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-20 17:05:14,827 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-20 17:05:15,205 WARN [Listener at localhost/36759] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 17:05:15,316 WARN [Listener at localhost/36759] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-20 17:05:15,334 WARN [Listener at localhost/36759] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 17:05:15,340 INFO [Listener at localhost/36759] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 17:05:15,344 INFO [Listener at localhost/36759] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/java.io.tmpdir/Jetty_localhost_46343_datanode____.aq3e95/webapp 2023-07-20 17:05:15,440 INFO [Listener at localhost/36759] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46343 2023-07-20 17:05:15,757 WARN [Listener at localhost/37789] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 17:05:15,767 WARN [Listener at localhost/37789] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-20 17:05:15,772 WARN [Listener at localhost/37789] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 17:05:15,774 INFO [Listener at localhost/37789] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 17:05:15,779 INFO [Listener at localhost/37789] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/java.io.tmpdir/Jetty_localhost_39757_datanode____disqje/webapp 2023-07-20 17:05:15,882 INFO [Listener at localhost/37789] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39757 2023-07-20 17:05:15,891 WARN [Listener at localhost/41543] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 17:05:16,223 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x59dce0abe9f0df06: Processing first storage report for DS-e9bc25ac-18e7-43fc-b035-91edba77e418 from datanode 5db548ba-844c-4201-b2b2-1954fb393813 2023-07-20 17:05:16,225 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x59dce0abe9f0df06: from storage DS-e9bc25ac-18e7-43fc-b035-91edba77e418 node DatanodeRegistration(127.0.0.1:33579, datanodeUuid=5db548ba-844c-4201-b2b2-1954fb393813, infoPort=33023, infoSecurePort=0, ipcPort=41543, storageInfo=lv=-57;cid=testClusterID;nsid=1340585843;c=1689872714220), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-20 17:05:16,225 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd5e14c3cbad35cc8: Processing first storage report for DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3 from datanode 6bdb3cad-7122-42c2-93fe-60972a5bafaa 2023-07-20 17:05:16,225 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd5e14c3cbad35cc8: from storage DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3 node DatanodeRegistration(127.0.0.1:37295, datanodeUuid=6bdb3cad-7122-42c2-93fe-60972a5bafaa, infoPort=35297, infoSecurePort=0, ipcPort=37789, storageInfo=lv=-57;cid=testClusterID;nsid=1340585843;c=1689872714220), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-20 17:05:16,225 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x59dce0abe9f0df06: Processing first storage report for DS-66dc1820-40e4-4dd3-b86f-e565c77c9285 from datanode 5db548ba-844c-4201-b2b2-1954fb393813 2023-07-20 17:05:16,225 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x59dce0abe9f0df06: from storage DS-66dc1820-40e4-4dd3-b86f-e565c77c9285 node DatanodeRegistration(127.0.0.1:33579, datanodeUuid=5db548ba-844c-4201-b2b2-1954fb393813, infoPort=33023, infoSecurePort=0, ipcPort=41543, storageInfo=lv=-57;cid=testClusterID;nsid=1340585843;c=1689872714220), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-20 17:05:16,225 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd5e14c3cbad35cc8: Processing first storage report for DS-ef7f6c57-fc43-4820-8b94-e14269518e24 from datanode 6bdb3cad-7122-42c2-93fe-60972a5bafaa 2023-07-20 17:05:16,226 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd5e14c3cbad35cc8: from storage DS-ef7f6c57-fc43-4820-8b94-e14269518e24 node DatanodeRegistration(127.0.0.1:37295, datanodeUuid=6bdb3cad-7122-42c2-93fe-60972a5bafaa, infoPort=35297, infoSecurePort=0, ipcPort=37789, storageInfo=lv=-57;cid=testClusterID;nsid=1340585843;c=1689872714220), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-20 17:05:16,309 DEBUG [Listener at localhost/41543] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67 2023-07-20 17:05:16,372 INFO [Listener at localhost/41543] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56/zookeeper_0, clientPort=57299, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-20 17:05:16,388 INFO [Listener at localhost/41543] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57299 2023-07-20 17:05:16,399 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:16,402 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:17,061 INFO [Listener at localhost/41543] util.FSUtils(471): Created version file at hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f with version=8 2023-07-20 17:05:17,061 INFO [Listener at localhost/41543] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/hbase-staging 2023-07-20 17:05:17,381 INFO [Listener at localhost/41543] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-20 17:05:17,842 INFO [Listener at localhost/41543] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 17:05:17,873 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:17,874 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:17,874 INFO [Listener at localhost/41543] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 17:05:17,874 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:17,874 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 17:05:18,014 INFO [Listener at localhost/41543] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 17:05:18,086 DEBUG [Listener at localhost/41543] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-20 17:05:18,181 INFO [Listener at localhost/41543] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39439 2023-07-20 17:05:18,191 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:18,193 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:18,213 INFO [Listener at localhost/41543] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39439 connecting to ZooKeeper ensemble=127.0.0.1:57299 2023-07-20 17:05:18,253 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:394390x0, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 17:05:18,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:39439-0x10183b1d5470000 connected 2023-07-20 17:05:18,279 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 17:05:18,280 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 17:05:18,283 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 17:05:18,291 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39439 2023-07-20 17:05:18,291 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39439 2023-07-20 17:05:18,291 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39439 2023-07-20 17:05:18,292 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39439 2023-07-20 17:05:18,292 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39439 2023-07-20 17:05:18,298 INFO [Listener at localhost/41543] master.HMaster(444): hbase.rootdir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f, hbase.cluster.distributed=false 2023-07-20 17:05:18,363 INFO [Listener at localhost/41543] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 17:05:18,363 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:18,363 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:18,364 INFO [Listener at localhost/41543] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 17:05:18,364 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:18,364 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 17:05:18,368 INFO [Listener at localhost/41543] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 17:05:18,371 INFO [Listener at localhost/41543] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33725 2023-07-20 17:05:18,373 INFO [Listener at localhost/41543] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 17:05:18,379 DEBUG [Listener at localhost/41543] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 17:05:18,380 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:18,382 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:18,383 INFO [Listener at localhost/41543] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33725 connecting to ZooKeeper ensemble=127.0.0.1:57299 2023-07-20 17:05:18,387 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:337250x0, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 17:05:18,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33725-0x10183b1d5470001 connected 2023-07-20 17:05:18,388 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 17:05:18,390 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 17:05:18,390 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 17:05:18,391 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33725 2023-07-20 17:05:18,391 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33725 2023-07-20 17:05:18,391 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33725 2023-07-20 17:05:18,393 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33725 2023-07-20 17:05:18,393 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33725 2023-07-20 17:05:18,406 INFO [Listener at localhost/41543] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 17:05:18,406 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:18,406 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:18,407 INFO [Listener at localhost/41543] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 17:05:18,407 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:18,407 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 17:05:18,407 INFO [Listener at localhost/41543] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 17:05:18,409 INFO [Listener at localhost/41543] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46429 2023-07-20 17:05:18,409 INFO [Listener at localhost/41543] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 17:05:18,410 DEBUG [Listener at localhost/41543] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 17:05:18,411 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:18,412 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:18,414 INFO [Listener at localhost/41543] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46429 connecting to ZooKeeper ensemble=127.0.0.1:57299 2023-07-20 17:05:18,418 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:464290x0, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 17:05:18,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:46429-0x10183b1d5470002 connected 2023-07-20 17:05:18,419 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 17:05:18,420 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 17:05:18,420 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 17:05:18,421 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46429 2023-07-20 17:05:18,421 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46429 2023-07-20 17:05:18,421 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46429 2023-07-20 17:05:18,422 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46429 2023-07-20 17:05:18,422 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46429 2023-07-20 17:05:18,423 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:18,432 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-20 17:05:18,433 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:18,453 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 17:05:18,453 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 17:05:18,453 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:18,453 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 17:05:18,454 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 17:05:18,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39439,1689872717206 from backup master directory 2023-07-20 17:05:18,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 17:05:18,459 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:18,459 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-20 17:05:18,459 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 17:05:18,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:18,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-20 17:05:18,463 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-20 17:05:18,547 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/hbase.id with ID: fa75bdc8-d06b-46a1-b06b-5b81a9d1aa77 2023-07-20 17:05:18,590 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:18,604 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:18,647 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x06b95531 to 127.0.0.1:57299 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 17:05:18,679 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@683c30a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 17:05:18,702 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-20 17:05:18,704 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-20 17:05:18,720 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-20 17:05:18,720 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-20 17:05:18,722 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-20 17:05:18,726 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-20 17:05:18,727 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 17:05:18,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/data/master/store-tmp 2023-07-20 17:05:18,793 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:18,793 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-20 17:05:18,793 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 17:05:18,793 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 17:05:18,793 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-20 17:05:18,793 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 17:05:18,793 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 17:05:18,794 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 17:05:18,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/WALs/jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:18,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39439%2C1689872717206, suffix=, logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/WALs/jenkins-hbase4.apache.org,39439,1689872717206, archiveDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/oldWALs, maxLogs=10 2023-07-20 17:05:18,868 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK] 2023-07-20 17:05:18,868 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK] 2023-07-20 17:05:18,876 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 17:05:18,946 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/WALs/jenkins-hbase4.apache.org,39439,1689872717206/jenkins-hbase4.apache.org%2C39439%2C1689872717206.1689872718825 2023-07-20 17:05:18,947 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK], DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK]] 2023-07-20 17:05:18,948 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:18,948 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:18,952 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 17:05:18,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 17:05:19,008 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-20 17:05:19,016 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-20 17:05:19,048 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-20 17:05:19,059 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:19,065 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-20 17:05:19,067 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-20 17:05:19,081 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 17:05:19,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 17:05:19,087 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74201350, jitterRate=0.10568627715110779}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 17:05:19,087 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 17:05:19,088 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-20 17:05:19,107 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-20 17:05:19,107 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-20 17:05:19,110 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-20 17:05:19,111 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-20 17:05:19,144 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-07-20 17:05:19,144 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-20 17:05:19,171 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-20 17:05:19,176 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-20 17:05:19,200 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-20 17:05:19,204 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-20 17:05:19,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-20 17:05:19,210 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-20 17:05:19,214 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-20 17:05:19,219 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:19,220 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-20 17:05:19,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-20 17:05:19,232 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-20 17:05:19,236 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 17:05:19,236 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 17:05:19,236 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 17:05:19,237 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:19,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,39439,1689872717206, sessionid=0x10183b1d5470000, setting cluster-up flag (Was=false) 2023-07-20 17:05:19,253 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:19,259 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-20 17:05:19,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:19,267 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:19,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-20 17:05:19,273 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:19,276 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.hbase-snapshot/.tmp 2023-07-20 17:05:19,326 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(951): ClusterId : fa75bdc8-d06b-46a1-b06b-5b81a9d1aa77 2023-07-20 17:05:19,326 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(951): ClusterId : fa75bdc8-d06b-46a1-b06b-5b81a9d1aa77 2023-07-20 17:05:19,331 DEBUG [RS:1;jenkins-hbase4:46429] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 17:05:19,331 DEBUG [RS:0;jenkins-hbase4:33725] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 17:05:19,338 DEBUG [RS:1;jenkins-hbase4:46429] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 17:05:19,338 DEBUG [RS:0;jenkins-hbase4:33725] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 17:05:19,338 DEBUG [RS:1;jenkins-hbase4:46429] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 17:05:19,338 DEBUG [RS:0;jenkins-hbase4:33725] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 17:05:19,341 DEBUG [RS:1;jenkins-hbase4:46429] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 17:05:19,342 DEBUG [RS:0;jenkins-hbase4:33725] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 17:05:19,343 DEBUG [RS:1;jenkins-hbase4:46429] zookeeper.ReadOnlyZKClient(139): Connect 0x68b7b995 to 127.0.0.1:57299 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 17:05:19,343 DEBUG [RS:0;jenkins-hbase4:33725] zookeeper.ReadOnlyZKClient(139): Connect 0x25ee5e17 to 127.0.0.1:57299 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 17:05:19,351 DEBUG [RS:1;jenkins-hbase4:46429] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66705af3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 17:05:19,352 DEBUG [RS:1;jenkins-hbase4:46429] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@624fd2e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 17:05:19,352 DEBUG [RS:0;jenkins-hbase4:33725] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b29b4da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 17:05:19,352 DEBUG [RS:0;jenkins-hbase4:33725] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f163567, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 17:05:19,382 DEBUG [RS:1;jenkins-hbase4:46429] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:46429 2023-07-20 17:05:19,385 DEBUG [RS:0;jenkins-hbase4:33725] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:33725 2023-07-20 17:05:19,386 INFO [RS:0;jenkins-hbase4:33725] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 17:05:19,386 INFO [RS:1;jenkins-hbase4:46429] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 17:05:19,387 INFO [RS:0;jenkins-hbase4:33725] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 17:05:19,387 INFO [RS:1;jenkins-hbase4:46429] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 17:05:19,387 DEBUG [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 17:05:19,387 DEBUG [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 17:05:19,390 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,39439,1689872717206 with isa=jenkins-hbase4.apache.org/172.31.14.131:46429, startcode=1689872718405 2023-07-20 17:05:19,390 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,39439,1689872717206 with isa=jenkins-hbase4.apache.org/172.31.14.131:33725, startcode=1689872718362 2023-07-20 17:05:19,408 DEBUG [RS:0;jenkins-hbase4:33725] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 17:05:19,408 DEBUG [RS:1;jenkins-hbase4:46429] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 17:05:19,420 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-20 17:05:19,430 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 17:05:19,430 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 17:05:19,430 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 17:05:19,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 17:05:19,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-20 17:05:19,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 17:05:19,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,432 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689872749432 2023-07-20 17:05:19,435 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-20 17:05:19,448 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-20 17:05:19,452 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-20 17:05:19,453 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-20 17:05:19,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-20 17:05:19,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-20 17:05:19,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-20 17:05:19,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-20 17:05:19,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,457 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-20 17:05:19,458 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-20 17:05:19,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-20 17:05:19,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-20 17:05:19,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-20 17:05:19,463 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-20 17:05:19,464 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689872719464,5,FailOnTimeoutGroup] 2023-07-20 17:05:19,465 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689872719465,5,FailOnTimeoutGroup] 2023-07-20 17:05:19,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-20 17:05:19,466 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,466 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,470 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49931, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 17:05:19,470 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41909, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 17:05:19,482 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39439] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:19,496 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-20 17:05:19,497 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-20 17:05:19,497 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f 2023-07-20 17:05:19,501 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39439] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:19,508 DEBUG [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f 2023-07-20 17:05:19,508 DEBUG [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f 2023-07-20 17:05:19,508 DEBUG [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:36759 2023-07-20 17:05:19,508 DEBUG [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:36759 2023-07-20 17:05:19,508 DEBUG [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 17:05:19,508 DEBUG [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 17:05:19,518 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:19,519 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:19,522 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 17:05:19,524 DEBUG [RS:0;jenkins-hbase4:33725] zookeeper.ZKUtil(162): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:19,524 WARN [RS:0;jenkins-hbase4:33725] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 17:05:19,524 INFO [RS:0;jenkins-hbase4:33725] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 17:05:19,525 DEBUG [RS:1;jenkins-hbase4:46429] zookeeper.ZKUtil(162): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:19,525 DEBUG [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1948): logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:19,525 WARN [RS:1;jenkins-hbase4:46429] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 17:05:19,525 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info 2023-07-20 17:05:19,525 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33725,1689872718362] 2023-07-20 17:05:19,525 INFO [RS:1;jenkins-hbase4:46429] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 17:05:19,525 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46429,1689872718405] 2023-07-20 17:05:19,526 DEBUG [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1948): logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:19,527 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 17:05:19,528 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:19,529 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 17:05:19,533 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/rep_barrier 2023-07-20 17:05:19,534 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 17:05:19,535 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:19,535 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 17:05:19,537 DEBUG [RS:0;jenkins-hbase4:33725] zookeeper.ZKUtil(162): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:19,537 DEBUG [RS:1;jenkins-hbase4:46429] zookeeper.ZKUtil(162): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:19,538 DEBUG [RS:0;jenkins-hbase4:33725] zookeeper.ZKUtil(162): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:19,538 DEBUG [RS:1;jenkins-hbase4:46429] zookeeper.ZKUtil(162): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:19,538 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table 2023-07-20 17:05:19,538 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 17:05:19,539 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:19,541 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740 2023-07-20 17:05:19,542 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740 2023-07-20 17:05:19,546 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 17:05:19,549 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 17:05:19,551 DEBUG [RS:0;jenkins-hbase4:33725] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 17:05:19,551 DEBUG [RS:1;jenkins-hbase4:46429] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 17:05:19,553 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 17:05:19,554 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65460274, jitterRate=-0.024565905332565308}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 17:05:19,554 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 17:05:19,554 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 17:05:19,554 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 17:05:19,554 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 17:05:19,554 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 17:05:19,554 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 17:05:19,555 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 17:05:19,555 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 17:05:19,560 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-20 17:05:19,560 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-20 17:05:19,561 INFO [RS:0;jenkins-hbase4:33725] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 17:05:19,561 INFO [RS:1;jenkins-hbase4:46429] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 17:05:19,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 17:05:19,582 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 17:05:19,584 INFO [RS:1;jenkins-hbase4:46429] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 17:05:19,584 INFO [RS:0;jenkins-hbase4:33725] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 17:05:19,585 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-20 17:05:19,588 INFO [RS:0;jenkins-hbase4:33725] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 17:05:19,588 INFO [RS:1;jenkins-hbase4:46429] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 17:05:19,589 INFO [RS:0;jenkins-hbase4:33725] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,589 INFO [RS:1;jenkins-hbase4:46429] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,590 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 17:05:19,590 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 17:05:19,597 INFO [RS:0;jenkins-hbase4:33725] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,597 INFO [RS:1;jenkins-hbase4:46429] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,598 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,598 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,598 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,598 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,598 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,598 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,598 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,598 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,598 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,598 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,599 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 17:05:19,599 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 17:05:19,599 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,599 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,599 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,599 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,599 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,599 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,599 DEBUG [RS:0;jenkins-hbase4:33725] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,599 DEBUG [RS:1;jenkins-hbase4:46429] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:19,601 INFO [RS:0;jenkins-hbase4:33725] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,601 INFO [RS:0;jenkins-hbase4:33725] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,601 INFO [RS:0;jenkins-hbase4:33725] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,601 INFO [RS:1;jenkins-hbase4:46429] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,601 INFO [RS:1;jenkins-hbase4:46429] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,601 INFO [RS:1;jenkins-hbase4:46429] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,618 INFO [RS:1;jenkins-hbase4:46429] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 17:05:19,618 INFO [RS:0;jenkins-hbase4:33725] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 17:05:19,620 INFO [RS:1;jenkins-hbase4:46429] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46429,1689872718405-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,620 INFO [RS:0;jenkins-hbase4:33725] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33725,1689872718362-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:19,640 INFO [RS:1;jenkins-hbase4:46429] regionserver.Replication(203): jenkins-hbase4.apache.org,46429,1689872718405 started 2023-07-20 17:05:19,640 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,46429,1689872718405, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46429, sessionid=0x10183b1d5470002 2023-07-20 17:05:19,640 INFO [RS:0;jenkins-hbase4:33725] regionserver.Replication(203): jenkins-hbase4.apache.org,33725,1689872718362 started 2023-07-20 17:05:19,641 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,33725,1689872718362, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33725, sessionid=0x10183b1d5470001 2023-07-20 17:05:19,641 DEBUG [RS:1;jenkins-hbase4:46429] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 17:05:19,641 DEBUG [RS:0;jenkins-hbase4:33725] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 17:05:19,641 DEBUG [RS:1;jenkins-hbase4:46429] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:19,641 DEBUG [RS:0;jenkins-hbase4:33725] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:19,642 DEBUG [RS:1;jenkins-hbase4:46429] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46429,1689872718405' 2023-07-20 17:05:19,642 DEBUG [RS:0;jenkins-hbase4:33725] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33725,1689872718362' 2023-07-20 17:05:19,643 DEBUG [RS:0;jenkins-hbase4:33725] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 17:05:19,642 DEBUG [RS:1;jenkins-hbase4:46429] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 17:05:19,643 DEBUG [RS:0;jenkins-hbase4:33725] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 17:05:19,643 DEBUG [RS:1;jenkins-hbase4:46429] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 17:05:19,644 DEBUG [RS:0;jenkins-hbase4:33725] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 17:05:19,644 DEBUG [RS:1;jenkins-hbase4:46429] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 17:05:19,644 DEBUG [RS:1;jenkins-hbase4:46429] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 17:05:19,644 DEBUG [RS:0;jenkins-hbase4:33725] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 17:05:19,644 DEBUG [RS:1;jenkins-hbase4:46429] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:19,644 DEBUG [RS:0;jenkins-hbase4:33725] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:19,645 DEBUG [RS:1;jenkins-hbase4:46429] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46429,1689872718405' 2023-07-20 17:05:19,645 DEBUG [RS:1;jenkins-hbase4:46429] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 17:05:19,645 DEBUG [RS:0;jenkins-hbase4:33725] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33725,1689872718362' 2023-07-20 17:05:19,645 DEBUG [RS:0;jenkins-hbase4:33725] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 17:05:19,645 DEBUG [RS:1;jenkins-hbase4:46429] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 17:05:19,646 DEBUG [RS:0;jenkins-hbase4:33725] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 17:05:19,646 DEBUG [RS:1;jenkins-hbase4:46429] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 17:05:19,646 DEBUG [RS:0;jenkins-hbase4:33725] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 17:05:19,646 INFO [RS:1;jenkins-hbase4:46429] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 17:05:19,646 INFO [RS:0;jenkins-hbase4:33725] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 17:05:19,647 INFO [RS:1;jenkins-hbase4:46429] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 17:05:19,647 INFO [RS:0;jenkins-hbase4:33725] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 17:05:19,737 DEBUG [jenkins-hbase4:39439] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-20 17:05:19,741 DEBUG [jenkins-hbase4:39439] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-20 17:05:19,747 DEBUG [jenkins-hbase4:39439] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 17:05:19,747 DEBUG [jenkins-hbase4:39439] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 17:05:19,747 DEBUG [jenkins-hbase4:39439] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 17:05:19,749 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33725,1689872718362, state=OPENING 2023-07-20 17:05:19,757 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-20 17:05:19,758 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:19,758 INFO [RS:0;jenkins-hbase4:33725] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33725%2C1689872718362, suffix=, logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362, archiveDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs, maxLogs=32 2023-07-20 17:05:19,758 INFO [RS:1;jenkins-hbase4:46429] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46429%2C1689872718405, suffix=, logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405, archiveDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs, maxLogs=32 2023-07-20 17:05:19,759 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 17:05:19,764 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33725,1689872718362}] 2023-07-20 17:05:19,785 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK] 2023-07-20 17:05:19,785 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK] 2023-07-20 17:05:19,785 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK] 2023-07-20 17:05:19,786 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK] 2023-07-20 17:05:19,794 INFO [RS:1;jenkins-hbase4:46429] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765 2023-07-20 17:05:19,794 INFO [RS:0;jenkins-hbase4:33725] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765 2023-07-20 17:05:19,794 DEBUG [RS:1;jenkins-hbase4:46429] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK], DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK]] 2023-07-20 17:05:19,798 DEBUG [RS:0;jenkins-hbase4:33725] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK], DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK]] 2023-07-20 17:05:19,948 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:19,951 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 17:05:19,954 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 17:05:19,965 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 17:05:19,965 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 17:05:19,969 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta, suffix=.meta, logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362, archiveDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs, maxLogs=32 2023-07-20 17:05:19,989 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK] 2023-07-20 17:05:19,989 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK] 2023-07-20 17:05:19,996 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta 2023-07-20 17:05:19,997 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK], DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK]] 2023-07-20 17:05:19,997 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:19,999 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 17:05:20,015 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 17:05:20,050 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 17:05:20,056 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 17:05:20,056 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:20,056 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 17:05:20,056 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 17:05:20,059 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 17:05:20,061 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info 2023-07-20 17:05:20,061 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info 2023-07-20 17:05:20,062 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 17:05:20,063 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:20,063 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 17:05:20,064 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/rep_barrier 2023-07-20 17:05:20,064 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/rep_barrier 2023-07-20 17:05:20,065 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 17:05:20,066 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:20,066 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 17:05:20,067 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table 2023-07-20 17:05:20,067 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table 2023-07-20 17:05:20,068 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 17:05:20,068 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:20,070 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740 2023-07-20 17:05:20,074 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740 2023-07-20 17:05:20,077 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 17:05:20,079 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 17:05:20,081 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64423822, jitterRate=-0.0400102436542511}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 17:05:20,081 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 17:05:20,090 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689872719941 2023-07-20 17:05:20,107 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 17:05:20,108 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 17:05:20,109 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33725,1689872718362, state=OPEN 2023-07-20 17:05:20,112 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 17:05:20,113 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 17:05:20,117 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-20 17:05:20,117 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33725,1689872718362 in 350 msec 2023-07-20 17:05:20,121 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-20 17:05:20,122 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 548 msec 2023-07-20 17:05:20,128 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 794 msec 2023-07-20 17:05:20,129 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689872720128, completionTime=-1 2023-07-20 17:05:20,129 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-20 17:05:20,129 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-20 17:05:20,197 DEBUG [hconnection-0x9511d02-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:20,201 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49966, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:20,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-20 17:05:20,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689872780218 2023-07-20 17:05:20,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689872840218 2023-07-20 17:05:20,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 88 msec 2023-07-20 17:05:20,242 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39439,1689872717206-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:20,243 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39439,1689872717206-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:20,243 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39439,1689872717206-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:20,246 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39439, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:20,246 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:20,252 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-20 17:05:20,261 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-20 17:05:20,262 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-20 17:05:20,272 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-20 17:05:20,275 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-20 17:05:20,277 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-20 17:05:20,300 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:20,302 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51 empty. 2023-07-20 17:05:20,303 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:20,303 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-20 17:05:20,332 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-20 17:05:20,334 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => ff374c4df33a87c27ad51fbe2c400e51, NAME => 'hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp 2023-07-20 17:05:20,353 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:20,353 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing ff374c4df33a87c27ad51fbe2c400e51, disabling compactions & flushes 2023-07-20 17:05:20,353 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:20,353 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:20,353 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. after waiting 0 ms 2023-07-20 17:05:20,353 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:20,354 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:20,354 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for ff374c4df33a87c27ad51fbe2c400e51: 2023-07-20 17:05:20,358 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-20 17:05:20,374 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689872720361"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689872720361"}]},"ts":"1689872720361"} 2023-07-20 17:05:20,402 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-20 17:05:20,404 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-20 17:05:20,408 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689872720404"}]},"ts":"1689872720404"} 2023-07-20 17:05:20,414 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-20 17:05:20,420 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-20 17:05:20,422 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 17:05:20,422 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 17:05:20,422 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 17:05:20,424 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ff374c4df33a87c27ad51fbe2c400e51, ASSIGN}] 2023-07-20 17:05:20,427 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ff374c4df33a87c27ad51fbe2c400e51, ASSIGN 2023-07-20 17:05:20,429 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=ff374c4df33a87c27ad51fbe2c400e51, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46429,1689872718405; forceNewPlan=false, retain=false 2023-07-20 17:05:20,582 INFO [jenkins-hbase4:39439] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-20 17:05:20,584 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ff374c4df33a87c27ad51fbe2c400e51, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:20,584 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689872720583"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689872720583"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689872720583"}]},"ts":"1689872720583"} 2023-07-20 17:05:20,588 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure ff374c4df33a87c27ad51fbe2c400e51, server=jenkins-hbase4.apache.org,46429,1689872718405}] 2023-07-20 17:05:20,742 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:20,742 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 17:05:20,745 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40462, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 17:05:20,750 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:20,751 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ff374c4df33a87c27ad51fbe2c400e51, NAME => 'hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:20,752 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:20,752 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:20,752 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:20,752 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:20,754 INFO [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:20,757 DEBUG [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/info 2023-07-20 17:05:20,757 DEBUG [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/info 2023-07-20 17:05:20,757 INFO [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ff374c4df33a87c27ad51fbe2c400e51 columnFamilyName info 2023-07-20 17:05:20,758 INFO [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] regionserver.HStore(310): Store=ff374c4df33a87c27ad51fbe2c400e51/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:20,760 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:20,760 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:20,766 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:20,771 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 17:05:20,772 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ff374c4df33a87c27ad51fbe2c400e51; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66247633, jitterRate=-0.01283334195613861}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 17:05:20,772 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ff374c4df33a87c27ad51fbe2c400e51: 2023-07-20 17:05:20,774 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51., pid=6, masterSystemTime=1689872720742 2023-07-20 17:05:20,779 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:20,779 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:20,780 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=ff374c4df33a87c27ad51fbe2c400e51, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:20,781 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689872720780"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689872720780"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689872720780"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689872720780"}]},"ts":"1689872720780"} 2023-07-20 17:05:20,788 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-20 17:05:20,788 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure ff374c4df33a87c27ad51fbe2c400e51, server=jenkins-hbase4.apache.org,46429,1689872718405 in 197 msec 2023-07-20 17:05:20,792 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-20 17:05:20,792 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=ff374c4df33a87c27ad51fbe2c400e51, ASSIGN in 364 msec 2023-07-20 17:05:20,793 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-20 17:05:20,794 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689872720793"}]},"ts":"1689872720793"} 2023-07-20 17:05:20,797 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-20 17:05:20,800 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-20 17:05:20,803 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 537 msec 2023-07-20 17:05:20,875 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-20 17:05:20,876 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-20 17:05:20,876 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:20,900 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:20,902 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40470, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:20,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-20 17:05:20,934 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-20 17:05:20,942 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 33 msec 2023-07-20 17:05:20,950 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-20 17:05:20,960 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-20 17:05:20,965 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-07-20 17:05:20,976 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-20 17:05:20,978 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-20 17:05:20,979 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.518sec 2023-07-20 17:05:20,981 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-20 17:05:20,983 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-20 17:05:20,983 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-20 17:05:20,984 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39439,1689872717206-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-20 17:05:20,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39439,1689872717206-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-20 17:05:20,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-20 17:05:21,032 DEBUG [Listener at localhost/41543] zookeeper.ReadOnlyZKClient(139): Connect 0x2a84a831 to 127.0.0.1:57299 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 17:05:21,037 DEBUG [Listener at localhost/41543] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d639341, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 17:05:21,051 DEBUG [hconnection-0x1a97b0f1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:21,063 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54720, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:21,073 INFO [Listener at localhost/41543] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:21,083 DEBUG [Listener at localhost/41543] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-20 17:05:21,085 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59030, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-20 17:05:21,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-20 17:05:21,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-20 17:05:21,102 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-20 17:05:21,104 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-20 17:05:21,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-20 17:05:21,107 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp/data/default/t/da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,107 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp/data/default/t/da051012dc01a99ec4c321e35c395da0 empty. 2023-07-20 17:05:21,109 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp/data/default/t/da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,109 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-20 17:05:21,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 17:05:21,129 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-20 17:05:21,131 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => da051012dc01a99ec4c321e35c395da0, NAME => 't,,1689872721094.da051012dc01a99ec4c321e35c395da0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/.tmp 2023-07-20 17:05:21,148 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689872721094.da051012dc01a99ec4c321e35c395da0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:21,148 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing da051012dc01a99ec4c321e35c395da0, disabling compactions & flushes 2023-07-20 17:05:21,148 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,148 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,148 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689872721094.da051012dc01a99ec4c321e35c395da0. after waiting 0 ms 2023-07-20 17:05:21,148 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,148 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,148 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for da051012dc01a99ec4c321e35c395da0: 2023-07-20 17:05:21,153 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-20 17:05:21,155 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689872721094.da051012dc01a99ec4c321e35c395da0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689872721155"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689872721155"}]},"ts":"1689872721155"} 2023-07-20 17:05:21,158 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-20 17:05:21,159 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-20 17:05:21,160 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689872721160"}]},"ts":"1689872721160"} 2023-07-20 17:05:21,162 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-20 17:05:21,167 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-20 17:05:21,168 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 17:05:21,169 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 17:05:21,169 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 17:05:21,169 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN}] 2023-07-20 17:05:21,171 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN 2023-07-20 17:05:21,172 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33725,1689872718362; forceNewPlan=false, retain=false 2023-07-20 17:05:21,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 17:05:21,323 INFO [jenkins-hbase4:39439] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-20 17:05:21,324 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=da051012dc01a99ec4c321e35c395da0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:21,324 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689872721094.da051012dc01a99ec4c321e35c395da0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689872721324"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689872721324"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689872721324"}]},"ts":"1689872721324"} 2023-07-20 17:05:21,328 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure da051012dc01a99ec4c321e35c395da0, server=jenkins-hbase4.apache.org,33725,1689872718362}] 2023-07-20 17:05:21,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 17:05:21,487 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => da051012dc01a99ec4c321e35c395da0, NAME => 't,,1689872721094.da051012dc01a99ec4c321e35c395da0.', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:21,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689872721094.da051012dc01a99ec4c321e35c395da0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:21,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,489 INFO [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,492 DEBUG [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/f 2023-07-20 17:05:21,492 DEBUG [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/f 2023-07-20 17:05:21,493 INFO [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da051012dc01a99ec4c321e35c395da0 columnFamilyName f 2023-07-20 17:05:21,494 INFO [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] regionserver.HStore(310): Store=da051012dc01a99ec4c321e35c395da0/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:21,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 17:05:21,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened da051012dc01a99ec4c321e35c395da0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67389058, jitterRate=0.00417521595954895}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 17:05:21,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for da051012dc01a99ec4c321e35c395da0: 2023-07-20 17:05:21,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689872721094.da051012dc01a99ec4c321e35c395da0., pid=11, masterSystemTime=1689872721481 2023-07-20 17:05:21,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,509 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=da051012dc01a99ec4c321e35c395da0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:21,509 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689872721094.da051012dc01a99ec4c321e35c395da0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689872721509"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689872721509"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689872721509"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689872721509"}]},"ts":"1689872721509"} 2023-07-20 17:05:21,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-20 17:05:21,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure da051012dc01a99ec4c321e35c395da0, server=jenkins-hbase4.apache.org,33725,1689872718362 in 185 msec 2023-07-20 17:05:21,518 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-20 17:05:21,518 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN in 346 msec 2023-07-20 17:05:21,519 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-20 17:05:21,519 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689872721519"}]},"ts":"1689872721519"} 2023-07-20 17:05:21,521 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-20 17:05:21,524 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-20 17:05:21,526 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 426 msec 2023-07-20 17:05:21,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 17:05:21,727 INFO [Listener at localhost/41543] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-20 17:05:21,727 DEBUG [Listener at localhost/41543] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-20 17:05:21,729 INFO [Listener at localhost/41543] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-20 17:05:21,733 INFO [Listener at localhost/41543] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-20 17:05:21,733 INFO [Listener at localhost/41543] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-20 17:05:21,733 INFO [Listener at localhost/41543] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-20 17:05:21,780 INFO [Listener at localhost/41543] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=204, ProcessCount=171, AvailableMemoryMB=5783 2023-07-20 17:05:21,782 INFO [Listener at localhost/41543] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-20 17:05:21,786 DEBUG [Listener at localhost/41543] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 17:05:21,787 DEBUG [Listener at localhost/41543] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,787 INFO [Listener at localhost/41543] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-20 17:05:21,787 INFO [Listener at localhost/41543] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-20 17:05:21,790 INFO [Listener at localhost/41543] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689872720793/Put/vlen=2/seqid=0} 2023-07-20 17:05:21,790 INFO [Listener at localhost/41543] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-20 17:05:21,790 INFO [Listener at localhost/41543] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51./info:regioninfo/1689872720780/Put/vlen=41/seqid=0, hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51./info:seqnumDuringOpen/1689872720780/Put/vlen=8/seqid=0, hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51./info:server/1689872720780/Put/vlen=31/seqid=0, hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51./info:serverstartcode/1689872720780/Put/vlen=8/seqid=0, hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51./info:sn/1689872720583/Put/vlen=45/seqid=0, hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51./info:state/1689872720780/Put/vlen=4/seqid=0} 2023-07-20 17:05:21,791 INFO [Listener at localhost/41543] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => ff374c4df33a87c27ad51fbe2c400e51, NAME => 'hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:21,791 INFO [Listener at localhost/41543] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689872721519/Put/vlen=2/seqid=0} 2023-07-20 17:05:21,791 INFO [Listener at localhost/41543] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-20 17:05:21,791 INFO [Listener at localhost/41543] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689872721094.da051012dc01a99ec4c321e35c395da0./info:regioninfo/1689872721509/Put/vlen=35/seqid=0, t,,1689872721094.da051012dc01a99ec4c321e35c395da0./info:seqnumDuringOpen/1689872721509/Put/vlen=8/seqid=0, t,,1689872721094.da051012dc01a99ec4c321e35c395da0./info:server/1689872721509/Put/vlen=31/seqid=0, t,,1689872721094.da051012dc01a99ec4c321e35c395da0./info:serverstartcode/1689872721509/Put/vlen=8/seqid=0, t,,1689872721094.da051012dc01a99ec4c321e35c395da0./info:sn/1689872721324/Put/vlen=45/seqid=0, t,,1689872721094.da051012dc01a99ec4c321e35c395da0./info:state/1689872721509/Put/vlen=4/seqid=0} 2023-07-20 17:05:21,791 INFO [Listener at localhost/41543] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => da051012dc01a99ec4c321e35c395da0, NAME => 't,,1689872721094.da051012dc01a99ec4c321e35c395da0.', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:21,792 DEBUG [Listener at localhost/41543] zookeeper.ReadOnlyZKClient(139): Connect 0x58f96b33 to 127.0.0.1:57299 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 17:05:21,827 DEBUG [Listener at localhost/41543] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f7b538a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 17:05:21,827 INFO [Listener at localhost/41543] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-20 17:05:21,827 INFO [Listener at localhost/41543] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-20 17:05:21,830 DEBUG [hconnection-0x6f3baee2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:21,833 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:21,839 INFO [Listener at localhost/41543] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-20 17:05:21,842 ERROR [Listener at localhost/41543] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,33725,1689872718362: die! ***** 2023-07-20 17:05:21,842 ERROR [Listener at localhost/41543] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-20 17:05:21,845 DEBUG [Listener at localhost/41543] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-20 17:05:21,848 DEBUG [Listener at localhost/41543] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-20 17:05:21,848 DEBUG [Listener at localhost/41543] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-20 17:05:21,849 DEBUG [Listener at localhost/41543] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-20 17:05:21,849 INFO [Listener at localhost/41543] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 981467136, "init": 513802240, "max": 2051014656, "used": 143963816 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 111053280 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-20 17:05:21,853 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39439] master.MasterRpcServices(612): jenkins-hbase4.apache.org,33725,1689872718362 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,33725,1689872718362: die! ***** 2023-07-20 17:05:21,855 INFO [Listener at localhost/41543] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,33725,1689872718362' ***** 2023-07-20 17:05:21,856 INFO [Listener at localhost/41543] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-20 17:05:21,856 INFO [RS:0;jenkins-hbase4:33725] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 17:05:21,856 INFO [RS:0;jenkins-hbase4:33725] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-20 17:05:21,856 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 17:05:21,857 INFO [RS:0;jenkins-hbase4:33725] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-20 17:05:21,857 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(3305): Received CLOSE for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:21,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33725] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -5128528694767255541 2023-07-20 17:05:21,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33725] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33725,1689872718362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 17:05:21,858 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:21,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33725] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:54732 deadline: 1689872781857, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33725,1689872718362 aborting 2023-07-20 17:05:21,858 DEBUG [RS:0;jenkins-hbase4:33725] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25ee5e17 to 127.0.0.1:57299 2023-07-20 17:05:21,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing da051012dc01a99ec4c321e35c395da0, disabling compactions & flushes 2023-07-20 17:05:21,858 DEBUG [RS:0;jenkins-hbase4:33725] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:21,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689872721094.da051012dc01a99ec4c321e35c395da0. after waiting 0 ms 2023-07-20 17:05:21,859 INFO [RS:0;jenkins-hbase4:33725] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 17:05:21,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,859 INFO [RS:0;jenkins-hbase4:33725] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 17:05:21,859 INFO [RS:0;jenkins-hbase4:33725] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 17:05:21,859 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 17:05:21,859 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for da051012dc01a99ec4c321e35c395da0: 2023-07-20 17:05:21,860 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-20 17:05:21,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:21,860 DEBUG [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-07-20 17:05:21,861 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 17:05:21,861 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 17:05:21,861 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 17:05:21,861 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 17:05:21,861 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 17:05:21,862 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-20 17:05:21,862 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 17:05:21,863 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 17:05:21,863 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 17:05:21,863 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 17:05:21,863 DEBUG [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1504): Waiting on 2023-07-20 17:05:21,877 DEBUG [Listener at localhost/41543] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33725,1689872718362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33725,1689872718362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-20 17:05:21,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33725] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33725,1689872718362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 17:05:21,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33725] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:54732 deadline: 1689872781879, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33725,1689872718362 aborting 2023-07-20 17:05:21,983 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33725] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33725,1689872718362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 17:05:21,983 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33725] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:54732 deadline: 1689872781983, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33725,1689872718362 aborting 2023-07-20 17:05:22,063 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,33725,1689872718362; all regions closed. 2023-07-20 17:05:22,079 DEBUG [RS:0;jenkins-hbase4:33725] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:22,080 INFO [RS:0;jenkins-hbase4:33725] regionserver.LeaseManager(133): Closed leases 2023-07-20 17:05:22,080 INFO [RS:0;jenkins-hbase4:33725] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-20 17:05:22,081 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 17:05:22,081 INFO [RS:0;jenkins-hbase4:33725] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33725 2023-07-20 17:05:22,089 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:33725 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedSocketException: Connection reset by peer: jenkins-hbase4.apache.org/172.31.14.131:33725 Caused by: java.net.SocketException: Connection reset by peer at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 17:05:22,091 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:33725 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedSocketException: Connection reset by peer: jenkins-hbase4.apache.org/172.31.14.131:33725 2023-07-20 17:05:22,091 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:22,091 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:22,091 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:22,091 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:22,091 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:22,093 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33725,1689872718362] 2023-07-20 17:05:22,093 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33725,1689872718362; numProcessing=1 2023-07-20 17:05:22,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:22,094 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,33725,1689872718362 znode expired, triggering replicatorRemoved event 2023-07-20 17:05:22,095 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33725,1689872718362 already deleted, retry=false 2023-07-20 17:05:22,095 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,33725,1689872718362 on jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:22,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:22,103 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,33725,1689872718362, splitWal=true, meta=true 2023-07-20 17:05:22,103 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,33725,1689872718362 (carryingMeta=true) jenkins-hbase4.apache.org,33725,1689872718362/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@11533cd1[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-20 17:05:22,106 INFO [PEWorker-3] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33725,1689872718362, splitWal=true, meta=true 2023-07-20 17:05:22,108 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33725,1689872718362, splitWal=true, meta=true, isMeta: true 2023-07-20 17:05:22,110 DEBUG [PEWorker-3] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting 2023-07-20 17:05:22,112 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,33725,1689872718362 WAL count=1, meta=true 2023-07-20 17:05:22,115 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta}] 2023-07-20 17:05:22,124 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:22,125 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta, worker=jenkins-hbase4.apache.org,46429,1689872718405}] 2023-07-20 17:05:22,193 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 17:05:22,194 INFO [RS:0;jenkins-hbase4:33725] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,33725,1689872718362; zookeeper connection closed. 2023-07-20 17:05:22,194 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:33725-0x10183b1d5470001, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 17:05:22,194 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4d1517c0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4d1517c0 2023-07-20 17:05:22,290 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46429] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-20 17:05:22,293 DEBUG [hconnection-0x6f3baee2-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:33725 this server is in the failed servers list 2023-07-20 17:05:22,312 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta, size=3.1 K (3185bytes) 2023-07-20 17:05:22,313 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta 2023-07-20 17:05:22,318 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta after 5ms 2023-07-20 17:05:22,324 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta took 13ms 2023-07-20 17:05:22,340 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-20 17:05:22,342 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta so closing down 2023-07-20 17:05:22,342 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 17:05:22,347 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-20 17:05:22,349 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta.temp 2023-07-20 17:05:22,350 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 17:05:22,361 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-20 17:05:22,363 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta.temp to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 17:05:22,364 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-20 17:05:22,364 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta, journal: Splitting hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta, size=3.1 K (3185bytes) at 1689872722312Finishing writing output for hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta so closing down at 1689872722342 (+30 ms)Creating recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta.temp at 1689872722349 (+7 ms)3 split writer threads finished at 1689872722351 (+2 ms)Closed recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689872722361 (+10 ms)Rename recovered edits hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta.temp to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689872722364 (+3 ms)Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689872722364 2023-07-20 17:05:22,364 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta 2023-07-20 17:05:22,367 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-20 17:05:22,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.HMaster(3641): Remote procedure done, pid=14 2023-07-20 17:05:22,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] wal.WALSplitUtil(141): Moved hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs 2023-07-20 17:05:22,380 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-20 17:05:22,380 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta, worker=jenkins-hbase4.apache.org,46429,1689872718405 in 252 msec 2023-07-20 17:05:22,382 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:22,385 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-20 17:05:22,385 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C33725%2C1689872718362.meta.1689872719971.meta, worker=jenkins-hbase4.apache.org,46429,1689872718405 in 269 msec 2023-07-20 17:05:22,388 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting dir is empty, no logs to split. 2023-07-20 17:05:22,388 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,33725,1689872718362 WAL count=0, meta=true 2023-07-20 17:05:22,388 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,33725,1689872718362 WAL splitting is done? wals=0, meta=true 2023-07-20 17:05:22,389 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 17:05:22,390 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 17:05:22,391 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 17:05:22,542 DEBUG [jenkins-hbase4:39439] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 17:05:22,542 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46429,1689872718405, state=OPENING 2023-07-20 17:05:22,545 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 17:05:22,545 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 17:05:22,545 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46429,1689872718405}] 2023-07-20 17:05:22,596 WARN [ReadOnlyZKClient-127.0.0.1:57299@0x58f96b33] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-20 17:05:22,596 DEBUG [hconnection-0x6f3baee2-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:22,600 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40484, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:22,600 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46429] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:40484 deadline: 1689872782600, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:22,602 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 17:05:22,704 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 17:05:22,704 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 17:05:22,706 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta, suffix=.meta, logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405, archiveDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs, maxLogs=32 2023-07-20 17:05:22,723 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK] 2023-07-20 17:05:22,725 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK] 2023-07-20 17:05:22,728 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta 2023-07-20 17:05:22,729 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK], DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK]] 2023-07-20 17:05:22,729 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:22,729 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 17:05:22,729 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 17:05:22,730 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 17:05:22,730 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 17:05:22,730 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:22,730 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 17:05:22,730 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 17:05:22,732 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 17:05:22,734 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info 2023-07-20 17:05:22,734 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info 2023-07-20 17:05:22,734 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 17:05:22,735 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:22,735 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 17:05:22,737 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/rep_barrier 2023-07-20 17:05:22,737 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/rep_barrier 2023-07-20 17:05:22,738 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 17:05:22,738 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:22,739 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 17:05:22,740 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table 2023-07-20 17:05:22,740 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table 2023-07-20 17:05:22,740 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 17:05:22,741 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:22,743 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740 2023-07-20 17:05:22,746 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740 2023-07-20 17:05:22,748 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 17:05:22,756 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 17:05:22,757 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-20 17:05:22,814 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/.tmp/info/91a289e679b64e45a719b59d3302bdb0 2023-07-20 17:05:22,867 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/.tmp/table/c4353ef1ef90470db8e955a974fa3aa1 2023-07-20 17:05:22,875 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/.tmp/info/91a289e679b64e45a719b59d3302bdb0 as hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info/91a289e679b64e45a719b59d3302bdb0 2023-07-20 17:05:22,884 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info/91a289e679b64e45a719b59d3302bdb0, entries=20, sequenceid=13, filesize=6.9 K 2023-07-20 17:05:22,886 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/.tmp/table/c4353ef1ef90470db8e955a974fa3aa1 as hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table/c4353ef1ef90470db8e955a974fa3aa1 2023-07-20 17:05:22,894 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table/c4353ef1ef90470db8e955a974fa3aa1, entries=4, sequenceid=13, filesize=4.7 K 2023-07-20 17:05:22,895 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 137ms, sequenceid=13, compaction requested=false; wal=null 2023-07-20 17:05:22,896 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-20 17:05:22,899 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 17:05:22,902 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 17:05:22,904 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 17:05:22,907 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-20 17:05:22,908 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70595951, jitterRate=0.05196164548397064}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 17:05:22,908 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 17:05:22,909 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689872722698 2023-07-20 17:05:22,912 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 17:05:22,912 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 17:05:22,913 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46429,1689872718405, state=OPEN 2023-07-20 17:05:22,914 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 17:05:22,914 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 17:05:22,917 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-20 17:05:22,917 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46429,1689872718405 in 369 msec 2023-07-20 17:05:22,919 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-20 17:05:22,919 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 528 msec 2023-07-20 17:05:22,920 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,33725,1689872718362 had 2 regions 2023-07-20 17:05:22,921 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33725,1689872718362, splitWal=true, meta=true, isMeta: false 2023-07-20 17:05:22,923 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,33725,1689872718362 WAL count=1, meta=false 2023-07-20 17:05:22,923 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765}] 2023-07-20 17:05:22,925 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:22,926 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765, worker=jenkins-hbase4.apache.org,46429,1689872718405}] 2023-07-20 17:05:23,079 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46429] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-20 17:05:23,093 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765, size=1.1 K (1148bytes) 2023-07-20 17:05:23,093 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765 2023-07-20 17:05:23,094 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765 after 1ms 2023-07-20 17:05:23,097 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765 took 4ms 2023-07-20 17:05:23,100 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for da051012dc01a99ec4c321e35c395da0: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-20 17:05:23,102 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765 so closing down 2023-07-20 17:05:23,102 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 17:05:23,106 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765.temp 2023-07-20 17:05:23,107 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 17:05:23,125 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-20 17:05:23,128 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765.temp to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000013 2023-07-20 17:05:23,128 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 31 ms; skipped=1; WAL=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-20 17:05:23,129 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765, journal: Splitting hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765, size=1.1 K (1148bytes) at 1689872723093Finishing writing output for hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765 so closing down at 1689872723102 (+9 ms)Creating recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765.temp at 1689872723106 (+4 ms)3 split writer threads finished at 1689872723107 (+1 ms)Closed recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689872723125 (+18 ms)Rename recovered edits hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765.temp to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000013 at 1689872723128 (+3 ms)Processed 11 edits across 1 Regions in 31 ms; skipped=1; WAL=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689872723129 (+1 ms) 2023-07-20 17:05:23,129 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765 2023-07-20 17:05:23,129 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-20 17:05:23,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.HMaster(3641): Remote procedure done, pid=18 2023-07-20 17:05:23,131 DEBUG [hconnection-0x6f3baee2-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:33725 this server is in the failed servers list 2023-07-20 17:05:23,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] wal.WALSplitUtil(141): Moved hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting/jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765 to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs 2023-07-20 17:05:23,136 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-20 17:05:23,136 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765, worker=jenkins-hbase4.apache.org,46429,1689872718405 in 207 msec 2023-07-20 17:05:23,137 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:23,140 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-20 17:05:23,140 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C33725%2C1689872718362.1689872719765, worker=jenkins-hbase4.apache.org,46429,1689872718405 in 214 msec 2023-07-20 17:05:23,142 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,33725,1689872718362-splitting dir is empty, no logs to split. 2023-07-20 17:05:23,142 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,33725,1689872718362 WAL count=0, meta=false 2023-07-20 17:05:23,142 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,33725,1689872718362 WAL splitting is done? wals=0, meta=false 2023-07-20 17:05:23,144 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN}] 2023-07-20 17:05:23,146 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN 2023-07-20 17:05:23,146 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 17:05:23,297 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=da051012dc01a99ec4c321e35c395da0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:23,297 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689872721094.da051012dc01a99ec4c321e35c395da0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689872723297"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689872723297"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689872723297"}]},"ts":"1689872723297"} 2023-07-20 17:05:23,298 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:33725 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33725 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 17:05:23,300 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:33725 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33725 2023-07-20 17:05:23,338 DEBUG [hconnection-0x6f3baee2-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:33725 this server is in the failed servers list 2023-07-20 17:05:23,404 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure da051012dc01a99ec4c321e35c395da0, server=jenkins-hbase4.apache.org,46429,1689872718405}] 2023-07-20 17:05:23,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:23,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => da051012dc01a99ec4c321e35c395da0, NAME => 't,,1689872721094.da051012dc01a99ec4c321e35c395da0.', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:23,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:23,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689872721094.da051012dc01a99ec4c321e35c395da0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:23,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:23,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:23,564 INFO [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:23,566 DEBUG [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/f 2023-07-20 17:05:23,566 DEBUG [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/f 2023-07-20 17:05:23,566 INFO [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da051012dc01a99ec4c321e35c395da0 columnFamilyName f 2023-07-20 17:05:23,567 INFO [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] regionserver.HStore(310): Store=da051012dc01a99ec4c321e35c395da0/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:23,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:23,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:23,572 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000013 2023-07-20 17:05:23,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000013 2023-07-20 17:05:23,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing da051012dc01a99ec4c321e35c395da0 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-20 17:05:23,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/.tmp/f/40e2ab6e048b411c95b4e536195cea4d 2023-07-20 17:05:23,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/.tmp/f/40e2ab6e048b411c95b4e536195cea4d as hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/f/40e2ab6e048b411c95b4e536195cea4d 2023-07-20 17:05:23,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/f/40e2ab6e048b411c95b4e536195cea4d, entries=10, sequenceid=13, filesize=4.8 K 2023-07-20 17:05:23,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for da051012dc01a99ec4c321e35c395da0 in 42ms, sequenceid=13, compaction requested=false; wal=null 2023-07-20 17:05:23,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-20 17:05:23,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/0000000000000000013 2023-07-20 17:05:23,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:23,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-20 17:05:23,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened da051012dc01a99ec4c321e35c395da0; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60341985, jitterRate=-0.10083435475826263}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 17:05:23,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for da051012dc01a99ec4c321e35c395da0: 2023-07-20 17:05:23,643 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689872721094.da051012dc01a99ec4c321e35c395da0., pid=20, masterSystemTime=1689872723556 2023-07-20 17:05:23,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:23,648 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:23,648 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=da051012dc01a99ec4c321e35c395da0, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:23,648 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689872721094.da051012dc01a99ec4c321e35c395da0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689872723648"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689872723648"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689872723648"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689872723648"}]},"ts":"1689872723648"} 2023-07-20 17:05:23,655 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-20 17:05:23,655 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure da051012dc01a99ec4c321e35c395da0, server=jenkins-hbase4.apache.org,46429,1689872718405 in 247 msec 2023-07-20 17:05:23,657 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-07-20 17:05:23,657 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,33725,1689872718362 after splitting done 2023-07-20 17:05:23,658 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN in 511 msec 2023-07-20 17:05:23,658 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,33725,1689872718362 from processing; numProcessing=0 2023-07-20 17:05:23,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,33725,1689872718362, splitWal=true, meta=true in 1.5610 sec 2023-07-20 17:05:23,676 DEBUG [Listener at localhost/41543] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58f96b33 to 127.0.0.1:57299 2023-07-20 17:05:23,676 DEBUG [Listener at localhost/41543] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:23,676 INFO [Listener at localhost/41543] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-20 17:05:23,690 INFO [Listener at localhost/41543] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=271 (was 294), OpenFileDescriptor=569 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 204), ProcessCount=171 (was 171), AvailableMemoryMB=5731 (was 5783) 2023-07-20 17:05:23,705 INFO [Listener at localhost/41543] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=271, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=204, ProcessCount=171, AvailableMemoryMB=5731 2023-07-20 17:05:23,706 INFO [Listener at localhost/41543] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:33725 2023-07-20 17:05:23,718 INFO [Listener at localhost/41543] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 17:05:23,719 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:23,719 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:23,719 INFO [Listener at localhost/41543] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 17:05:23,719 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 17:05:23,719 INFO [Listener at localhost/41543] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 17:05:23,719 INFO [Listener at localhost/41543] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 17:05:23,721 INFO [Listener at localhost/41543] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38295 2023-07-20 17:05:23,722 INFO [Listener at localhost/41543] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 17:05:23,723 DEBUG [Listener at localhost/41543] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 17:05:23,724 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:23,726 INFO [Listener at localhost/41543] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 17:05:23,727 INFO [Listener at localhost/41543] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38295 connecting to ZooKeeper ensemble=127.0.0.1:57299 2023-07-20 17:05:23,730 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:382950x0, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 17:05:23,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:38295-0x10183b1d5470008 connected 2023-07-20 17:05:23,732 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(162): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 17:05:23,733 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(162): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-20 17:05:23,734 DEBUG [Listener at localhost/41543] zookeeper.ZKUtil(164): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 17:05:23,735 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38295 2023-07-20 17:05:23,736 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38295 2023-07-20 17:05:23,739 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38295 2023-07-20 17:05:23,740 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38295 2023-07-20 17:05:23,740 DEBUG [Listener at localhost/41543] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38295 2023-07-20 17:05:23,743 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(951): ClusterId : fa75bdc8-d06b-46a1-b06b-5b81a9d1aa77 2023-07-20 17:05:23,743 DEBUG [RS:2;jenkins-hbase4:38295] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 17:05:23,746 DEBUG [RS:2;jenkins-hbase4:38295] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 17:05:23,746 DEBUG [RS:2;jenkins-hbase4:38295] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 17:05:23,748 DEBUG [RS:2;jenkins-hbase4:38295] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 17:05:23,749 DEBUG [RS:2;jenkins-hbase4:38295] zookeeper.ReadOnlyZKClient(139): Connect 0x6cdca08e to 127.0.0.1:57299 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 17:05:23,754 DEBUG [RS:2;jenkins-hbase4:38295] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@621b5365, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 17:05:23,754 DEBUG [RS:2;jenkins-hbase4:38295] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6624cbfe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 17:05:23,762 DEBUG [RS:2;jenkins-hbase4:38295] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:38295 2023-07-20 17:05:23,762 INFO [RS:2;jenkins-hbase4:38295] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 17:05:23,763 INFO [RS:2;jenkins-hbase4:38295] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 17:05:23,763 DEBUG [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 17:05:23,763 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,39439,1689872717206 with isa=jenkins-hbase4.apache.org/172.31.14.131:38295, startcode=1689872723718 2023-07-20 17:05:23,764 DEBUG [RS:2;jenkins-hbase4:38295] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 17:05:23,766 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46095, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 17:05:23,767 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39439] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:23,767 DEBUG [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f 2023-07-20 17:05:23,767 DEBUG [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:36759 2023-07-20 17:05:23,768 DEBUG [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 17:05:23,769 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:23,769 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:23,770 DEBUG [RS:2;jenkins-hbase4:38295] zookeeper.ZKUtil(162): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:23,770 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38295,1689872723718] 2023-07-20 17:05:23,770 WARN [RS:2;jenkins-hbase4:38295] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 17:05:23,770 INFO [RS:2;jenkins-hbase4:38295] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 17:05:23,770 DEBUG [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1948): logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:23,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:23,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:23,779 DEBUG [RS:2;jenkins-hbase4:38295] zookeeper.ZKUtil(162): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:23,779 DEBUG [RS:2;jenkins-hbase4:38295] zookeeper.ZKUtil(162): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:23,781 DEBUG [RS:2;jenkins-hbase4:38295] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 17:05:23,781 INFO [RS:2;jenkins-hbase4:38295] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 17:05:23,784 INFO [RS:2;jenkins-hbase4:38295] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 17:05:23,784 INFO [RS:2;jenkins-hbase4:38295] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 17:05:23,784 INFO [RS:2;jenkins-hbase4:38295] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:23,785 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 17:05:23,786 INFO [RS:2;jenkins-hbase4:38295] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:23,787 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:23,787 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:23,787 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:23,787 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:23,787 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:23,787 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 17:05:23,787 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:23,787 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:23,788 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:23,788 DEBUG [RS:2;jenkins-hbase4:38295] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 17:05:23,789 INFO [RS:2;jenkins-hbase4:38295] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:23,790 INFO [RS:2;jenkins-hbase4:38295] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:23,790 INFO [RS:2;jenkins-hbase4:38295] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:23,802 INFO [RS:2;jenkins-hbase4:38295] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 17:05:23,802 INFO [RS:2;jenkins-hbase4:38295] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38295,1689872723718-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 17:05:23,814 INFO [RS:2;jenkins-hbase4:38295] regionserver.Replication(203): jenkins-hbase4.apache.org,38295,1689872723718 started 2023-07-20 17:05:23,814 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,38295,1689872723718, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38295, sessionid=0x10183b1d5470008 2023-07-20 17:05:23,814 INFO [Listener at localhost/41543] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:38295,5,FailOnTimeoutGroup] 2023-07-20 17:05:23,814 DEBUG [RS:2;jenkins-hbase4:38295] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 17:05:23,814 DEBUG [RS:2;jenkins-hbase4:38295] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:23,814 DEBUG [RS:2;jenkins-hbase4:38295] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38295,1689872723718' 2023-07-20 17:05:23,815 DEBUG [RS:2;jenkins-hbase4:38295] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 17:05:23,814 INFO [Listener at localhost/41543] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:23,815 INFO [Listener at localhost/41543] client.TestScannerTimeout(177): START ************ test3686b 2023-07-20 17:05:23,815 DEBUG [RS:2;jenkins-hbase4:38295] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 17:05:23,816 DEBUG [RS:2;jenkins-hbase4:38295] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 17:05:23,816 DEBUG [RS:2;jenkins-hbase4:38295] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 17:05:23,816 DEBUG [RS:2;jenkins-hbase4:38295] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:23,816 DEBUG [RS:2;jenkins-hbase4:38295] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38295,1689872723718' 2023-07-20 17:05:23,816 DEBUG [RS:2;jenkins-hbase4:38295] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 17:05:23,816 DEBUG [RS:2;jenkins-hbase4:38295] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 17:05:23,817 DEBUG [RS:2;jenkins-hbase4:38295] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 17:05:23,817 INFO [RS:2;jenkins-hbase4:38295] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 17:05:23,817 INFO [RS:2;jenkins-hbase4:38295] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 17:05:23,817 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:33725 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33725 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 17:05:23,818 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:33725 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33725 2023-07-20 17:05:23,919 INFO [RS:2;jenkins-hbase4:38295] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38295%2C1689872723718, suffix=, logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,38295,1689872723718, archiveDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs, maxLogs=32 2023-07-20 17:05:23,920 DEBUG [hconnection-0x1a97b0f1-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:23,923 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40500, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:23,927 DEBUG [Listener at localhost/41543] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 17:05:23,927 DEBUG [Listener at localhost/41543] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:23,927 DEBUG [Listener at localhost/41543] zookeeper.ReadOnlyZKClient(139): Connect 0x5ac48154 to 127.0.0.1:57299 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 17:05:23,932 DEBUG [Listener at localhost/41543] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44c30efb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 17:05:23,935 DEBUG [hconnection-0x37cccf27-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:23,939 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40512, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:23,948 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK] 2023-07-20 17:05:23,949 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK] 2023-07-20 17:05:23,955 INFO [RS:2;jenkins-hbase4:38295] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,38295,1689872723718/jenkins-hbase4.apache.org%2C38295%2C1689872723718.1689872723921 2023-07-20 17:05:23,955 DEBUG [RS:2;jenkins-hbase4:38295] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK], DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK]] 2023-07-20 17:05:25,568 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-20 17:05:25,630 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-20 17:05:27,376 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 17:05:27,377 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-20 17:05:38,949 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 887674971629813768 expired clientIPAndPort=172.31.14.131:40512, userName=jenkins, regionInfo=t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:40,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46429] regionserver.RSRpcServices(3094): Client tried to access missing scanner 887674971629813768 2023-07-20 17:05:40,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46429] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:40512 deadline: 1689872800951, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '887674971629813768'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-20 17:05:40,952 DEBUG [Listener at localhost/41543] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '887674971629813768'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '887674971629813768'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '887674971629813768'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-20 17:05:40,956 DEBUG [Listener at localhost/41543] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ac48154 to 127.0.0.1:57299 2023-07-20 17:05:40,956 DEBUG [Listener at localhost/41543] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:40,956 INFO [Listener at localhost/41543] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-20 17:05:40,972 INFO [Listener at localhost/41543] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=296 (was 271) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f-prefix:jenkins-hbase4.apache.org,38295,1689872723718 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:2;jenkins-hbase4:38295-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-107507886-172.31.14.131-1689872714220:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57299@0x6cdca08e sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/829671167.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1471980821_17 at /127.0.0.1:39792 [Receiving block BP-107507886-172.31.14.131-1689872714220:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase4:38295Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-107507886-172.31.14.131-1689872714220:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1471980821_17 at /127.0.0.1:57252 [Receiving block BP-107507886-172.31.14.131-1689872714220:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57299@0x5ac48154-SendThread(127.0.0.1:57299) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57299@0x5ac48154 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/829671167.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57299@0x6cdca08e-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57299@0x5ac48154-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:38295 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57299@0x6cdca08e-SendThread(127.0.0.1:57299) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:36759 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38295 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=573 (was 569) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=175 (was 204), ProcessCount=171 (was 171), AvailableMemoryMB=6077 (was 5731) - AvailableMemoryMB LEAK? - 2023-07-20 17:05:40,991 INFO [Listener at localhost/41543] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=296, OpenFileDescriptor=573, MaxFileDescriptor=60000, SystemLoadAverage=175, ProcessCount=171, AvailableMemoryMB=6075 2023-07-20 17:05:40,991 INFO [Listener at localhost/41543] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:33725 2023-07-20 17:05:40,991 INFO [Listener at localhost/41543] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,33725,1689872718362 2023-07-20 17:05:40,992 INFO [Listener at localhost/41543] client.TestScannerTimeout(104): START************ test2772 2023-07-20 17:05:40,996 DEBUG [Listener at localhost/41543] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 17:05:40,996 DEBUG [Listener at localhost/41543] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:40,997 DEBUG [Listener at localhost/41543] zookeeper.ReadOnlyZKClient(139): Connect 0x003d8b32 to 127.0.0.1:57299 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 17:05:41,002 DEBUG [Listener at localhost/41543] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d6e31d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 17:05:41,002 ERROR [Listener at localhost/41543] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,46429,1689872718405: die! ***** 2023-07-20 17:05:41,002 ERROR [Listener at localhost/41543] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-20 17:05:41,003 DEBUG [Listener at localhost/41543] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-20 17:05:41,003 DEBUG [Listener at localhost/41543] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-20 17:05:41,009 DEBUG [Listener at localhost/41543] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-20 17:05:41,012 DEBUG [Listener at localhost/41543] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-20 17:05:41,030 INFO [Listener at localhost/41543] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 981467136, "init": 513802240, "max": 2051014656, "used": 276316808 }, "NonHeapMemoryUsage": { "committed": 123953152, "init": 2555904, "max": -1, "used": 121618360 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 12589, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 5, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16920 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:57299", "tag.serverName": "jenkins-hbase4.apache.org,33725,1689872718362", "tag.clusterId": "fa75bdc8-d06b-46a1-b06b-5b81a9d1aa77", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689872718362, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1787, "localBytesRead": 1787, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 7076, "ByteBuffAllocatorPoolAllocationBytes": 665600, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 33, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 2, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 11, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 20, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 11, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 20, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 2, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 11, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 52, "ServerReadQueryPerSecond_mean_rate": 2.424947603640661, "ServerReadQueryPerSecond_1min_rate": 7.164967204256925, "ServerReadQueryPerSecond_5min_rate": 8.751310705406569, "ServerReadQueryPerSecond_15min_rate": 9.047937375158883, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.1192067891685218, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-20 17:05:41,031 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39439] master.MasterRpcServices(612): jenkins-hbase4.apache.org,46429,1689872718405 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,46429,1689872718405: die! ***** 2023-07-20 17:05:41,034 INFO [Listener at localhost/41543] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,46429,1689872718405' ***** 2023-07-20 17:05:41,035 INFO [Listener at localhost/41543] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-20 17:05:41,035 INFO [RS:1;jenkins-hbase4:46429] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 17:05:41,035 INFO [RS:1;jenkins-hbase4:46429] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-20 17:05:41,035 INFO [RS:1;jenkins-hbase4:46429] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-20 17:05:41,036 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(3305): Received CLOSE for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:41,040 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 17:05:41,040 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(3305): Received CLOSE for ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:41,040 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:41,040 DEBUG [RS:1;jenkins-hbase4:46429] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68b7b995 to 127.0.0.1:57299 2023-07-20 17:05:41,041 DEBUG [RS:1;jenkins-hbase4:46429] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:41,041 INFO [RS:1;jenkins-hbase4:46429] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 17:05:41,041 INFO [RS:1;jenkins-hbase4:46429] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 17:05:41,041 INFO [RS:1;jenkins-hbase4:46429] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 17:05:41,041 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 17:05:41,042 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 17:05:41,042 DEBUG [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1478): Online Regions={da051012dc01a99ec4c321e35c395da0=t,,1689872721094.da051012dc01a99ec4c321e35c395da0., 1588230740=hbase:meta,,1.1588230740, ff374c4df33a87c27ad51fbe2c400e51=hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.} 2023-07-20 17:05:41,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing da051012dc01a99ec4c321e35c395da0, disabling compactions & flushes 2023-07-20 17:05:41,043 DEBUG [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1504): Waiting on 1588230740, da051012dc01a99ec4c321e35c395da0, ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:41,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:41,043 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:41,044 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 17:05:41,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:41,044 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 17:05:41,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689872721094.da051012dc01a99ec4c321e35c395da0. after waiting 0 ms 2023-07-20 17:05:41,044 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 17:05:41,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:41,044 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 17:05:41,044 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 17:05:41,049 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:41,049 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 834 in region hbase:meta,,1.1588230740 2023-07-20 17:05:41,049 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 17:05:41,049 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46429] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46429,1689872718405 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 17:05:41,050 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46429] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:44202 deadline: 1689872801049, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46429,1689872718405 aborting 2023-07-20 17:05:41,050 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 17:05:41,050 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 17:05:41,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:41,051 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 17:05:41,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for da051012dc01a99ec4c321e35c395da0: 2023-07-20 17:05:41,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:41,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ff374c4df33a87c27ad51fbe2c400e51, disabling compactions & flushes 2023-07-20 17:05:41,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:41,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:41,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. after waiting 0 ms 2023-07-20 17:05:41,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:41,055 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:41,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:41,055 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ff374c4df33a87c27ad51fbe2c400e51: 2023-07-20 17:05:41,055 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:41,153 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46429] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46429,1689872718405 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 17:05:41,153 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46429] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:44202 deadline: 1689872801153, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46429,1689872718405 aborting 2023-07-20 17:05:41,243 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,46429,1689872718405; all regions closed. 2023-07-20 17:05:41,257 DEBUG [RS:1;jenkins-hbase4:46429] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:41,257 INFO [RS:1;jenkins-hbase4:46429] regionserver.LeaseManager(133): Closed leases 2023-07-20 17:05:41,257 INFO [RS:1;jenkins-hbase4:46429] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-20 17:05:41,258 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 17:05:41,259 INFO [RS:1;jenkins-hbase4:46429] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46429 2023-07-20 17:05:41,263 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:41,263 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:41,263 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 2023-07-20 17:05:41,263 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:41,271 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:41,273 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46429,1689872718405] 2023-07-20 17:05:41,274 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46429,1689872718405; numProcessing=1 2023-07-20 17:05:41,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:41,274 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,46429,1689872718405 znode expired, triggering replicatorRemoved event 2023-07-20 17:05:41,275 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46429,1689872718405 already deleted, retry=false 2023-07-20 17:05:41,275 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,46429,1689872718405 on jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:41,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:41,276 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=21, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,46429,1689872718405, splitWal=true, meta=true 2023-07-20 17:05:41,277 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=21 for jenkins-hbase4.apache.org,46429,1689872718405 (carryingMeta=true) jenkins-hbase4.apache.org,46429,1689872718405/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@249058f7[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-20 17:05:41,278 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=21, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46429,1689872718405, splitWal=true, meta=true 2023-07-20 17:05:41,279 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46429,1689872718405, splitWal=true, meta=true, isMeta: true 2023-07-20 17:05:41,284 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting 2023-07-20 17:05:41,285 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,46429,1689872718405 WAL count=1, meta=true 2023-07-20 17:05:41,286 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta}] 2023-07-20 17:05:41,288 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:41,290 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta, worker=jenkins-hbase4.apache.org,38295,1689872723718}] 2023-07-20 17:05:41,356 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:46429 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46429 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 17:05:41,357 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:46429 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46429 2023-07-20 17:05:41,372 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 17:05:41,374 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 17:05:41,374 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:46429-0x10183b1d5470002, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 17:05:41,374 INFO [RS:1;jenkins-hbase4:46429] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,46429,1689872718405; zookeeper connection closed. 2023-07-20 17:05:41,374 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@16e58612] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@16e58612 2023-07-20 17:05:41,443 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:41,443 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 17:05:41,444 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34042, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 17:05:41,445 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38295] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-20 17:05:41,461 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta, size=1.3 K (1341bytes) 2023-07-20 17:05:41,461 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta 2023-07-20 17:05:41,462 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta after 0ms 2023-07-20 17:05:41,468 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta took 7ms 2023-07-20 17:05:41,473 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-20 17:05:41,474 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta so closing down 2023-07-20 17:05:41,474 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 17:05:41,478 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta.temp 2023-07-20 17:05:41,479 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 17:05:41,491 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-20 17:05:41,493 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta.temp to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-20 17:05:41,494 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 3 edits across 1 Regions in 26 ms; skipped=1; WAL=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false 2023-07-20 17:05:41,494 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta, journal: Splitting hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta, size=1.3 K (1341bytes) at 1689872741461Finishing writing output for hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta so closing down at 1689872741474 (+13 ms)Creating recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta.temp at 1689872741479 (+5 ms)3 split writer threads finished at 1689872741479Closed recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689872741491 (+12 ms)Rename recovered edits hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta.temp to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 at 1689872741493 (+2 ms)Processed 3 edits across 1 Regions in 26 ms; skipped=1; WAL=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false at 1689872741494 (+1 ms) 2023-07-20 17:05:41,494 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta 2023-07-20 17:05:41,494 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-20 17:05:41,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.HMaster(3641): Remote procedure done, pid=23 2023-07-20 17:05:41,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] wal.WALSplitUtil(141): Moved hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs 2023-07-20 17:05:41,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-20 17:05:41,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta, worker=jenkins-hbase4.apache.org,38295,1689872723718 in 210 msec 2023-07-20 17:05:41,502 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:41,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-07-20 17:05:41,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C46429%2C1689872718405.meta.1689872722708.meta, worker=jenkins-hbase4.apache.org,38295,1689872723718 in 216 msec 2023-07-20 17:05:41,507 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting dir is empty, no logs to split. 2023-07-20 17:05:41,507 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,46429,1689872718405 WAL count=0, meta=true 2023-07-20 17:05:41,507 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,46429,1689872718405 WAL splitting is done? wals=0, meta=true 2023-07-20 17:05:41,508 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 17:05:41,509 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 17:05:41,510 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 17:05:41,660 DEBUG [jenkins-hbase4:39439] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 17:05:41,660 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38295,1689872723718, state=OPENING 2023-07-20 17:05:41,660 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46429 this server is in the failed servers list 2023-07-20 17:05:41,665 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 17:05:41,665 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 17:05:41,665 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38295,1689872723718}] 2023-07-20 17:05:41,825 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 17:05:41,825 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 17:05:41,828 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38295%2C1689872723718.meta, suffix=.meta, logDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,38295,1689872723718, archiveDir=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs, maxLogs=32 2023-07-20 17:05:41,847 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK] 2023-07-20 17:05:41,847 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK] 2023-07-20 17:05:41,850 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,38295,1689872723718/jenkins-hbase4.apache.org%2C38295%2C1689872723718.meta.1689872741829.meta 2023-07-20 17:05:41,850 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37295,DS-513bb011-0af9-4ca6-a5f1-44e77d8258a3,DISK], DatanodeInfoWithStorage[127.0.0.1:33579,DS-e9bc25ac-18e7-43fc-b035-91edba77e418,DISK]] 2023-07-20 17:05:41,851 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:41,851 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 17:05:41,851 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 17:05:41,851 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 17:05:41,851 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 17:05:41,851 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:41,851 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 17:05:41,851 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 17:05:41,855 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 17:05:41,856 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info 2023-07-20 17:05:41,856 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info 2023-07-20 17:05:41,857 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 17:05:41,868 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info/91a289e679b64e45a719b59d3302bdb0 2023-07-20 17:05:41,869 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:41,869 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 17:05:41,870 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/rep_barrier 2023-07-20 17:05:41,871 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/rep_barrier 2023-07-20 17:05:41,871 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 17:05:41,872 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:41,872 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 17:05:41,873 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table 2023-07-20 17:05:41,873 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table 2023-07-20 17:05:41,875 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 17:05:41,885 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/table/c4353ef1ef90470db8e955a974fa3aa1 2023-07-20 17:05:41,885 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:41,886 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740 2023-07-20 17:05:41,888 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740 2023-07-20 17:05:41,889 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-20 17:05:41,895 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 8, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=17, path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-20 17:05:41,895 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=834 B heapSize=2.16 KB 2023-07-20 17:05:41,918 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=834 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/.tmp/info/1420ab1657224718af66cd16c9b3a876 2023-07-20 17:05:41,927 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/.tmp/info/1420ab1657224718af66cd16c9b3a876 as hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info/1420ab1657224718af66cd16c9b3a876 2023-07-20 17:05:41,940 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info/1420ab1657224718af66cd16c9b3a876, entries=8, sequenceid=17, filesize=5.5 K 2023-07-20 17:05:41,941 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~834 B/834, heapSize ~1.64 KB/1680, currentSize=0 B/0 for 1588230740 in 46ms, sequenceid=17, compaction requested=false; wal=null 2023-07-20 17:05:41,942 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-20 17:05:41,946 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 17:05:41,948 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 17:05:41,950 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=13 2023-07-20 17:05:41,952 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=18; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60481943, jitterRate=-0.0987488180398941}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 17:05:41,952 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 17:05:41,955 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=25, masterSystemTime=1689872741818 2023-07-20 17:05:41,961 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 17:05:41,962 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 17:05:41,963 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38295,1689872723718, state=OPEN 2023-07-20 17:05:41,964 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 17:05:41,964 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 17:05:41,968 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-20 17:05:41,968 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38295,1689872723718 in 299 msec 2023-07-20 17:05:41,975 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-20 17:05:41,975 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,46429,1689872718405 had 3 regions 2023-07-20 17:05:41,975 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 460 msec 2023-07-20 17:05:41,976 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46429,1689872718405, splitWal=true, meta=true, isMeta: false 2023-07-20 17:05:41,979 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,46429,1689872718405 WAL count=1, meta=false 2023-07-20 17:05:41,979 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765}] 2023-07-20 17:05:41,982 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:41,983 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765, worker=jenkins-hbase4.apache.org,38295,1689872723718}] 2023-07-20 17:05:42,137 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38295] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=27 2023-07-20 17:05:42,152 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765, size=899 (899bytes) 2023-07-20 17:05:42,152 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765 2023-07-20 17:05:42,153 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765 after 1ms 2023-07-20 17:05:42,157 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765 took 5ms 2023-07-20 17:05:42,160 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for ff374c4df33a87c27ad51fbe2c400e51: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-20 17:05:42,163 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for da051012dc01a99ec4c321e35c395da0: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-20 17:05:42,163 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765 so closing down 2023-07-20 17:05:42,163 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 17:05:42,166 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:42,166 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765.temp 2023-07-20 17:05:42,166 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 17:05:42,169 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:42,178 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46429 this server is in the failed servers list 2023-07-20 17:05:42,179 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-20 17:05:42,181 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765.temp to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000005 2023-07-20 17:05:42,181 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 24 ms; skipped=2; WAL=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765, size=899, length=899, corrupted=false, cancelled=false 2023-07-20 17:05:42,181 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765, journal: Splitting hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765, size=899 (899bytes) at 1689872742152Finishing writing output for hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765 so closing down at 1689872742163 (+11 ms)Creating recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765.temp at 1689872742166 (+3 ms)3 split writer threads finished at 1689872742166Closed recovered edits writer path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689872742180 (+14 ms)Rename recovered edits hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765.temp to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000005 at 1689872742181 (+1 ms)Processed 4 edits across 1 Regions in 24 ms; skipped=2; WAL=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765, size=899, length=899, corrupted=false, cancelled=false at 1689872742181 2023-07-20 17:05:42,182 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765 2023-07-20 17:05:42,182 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=27 2023-07-20 17:05:42,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] master.HMaster(3641): Remote procedure done, pid=27 2023-07-20 17:05:42,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39439] wal.WALSplitUtil(141): Moved hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting/jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765 to hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs 2023-07-20 17:05:42,188 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-20 17:05:42,188 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765, worker=jenkins-hbase4.apache.org,38295,1689872723718 in 203 msec 2023-07-20 17:05:42,190 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:42,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=21 2023-07-20 17:05:42,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C46429%2C1689872718405.1689872719765, worker=jenkins-hbase4.apache.org,38295,1689872723718 in 211 msec 2023-07-20 17:05:42,194 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/WALs/jenkins-hbase4.apache.org,46429,1689872718405-splitting dir is empty, no logs to split. 2023-07-20 17:05:42,194 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,46429,1689872718405 WAL count=0, meta=false 2023-07-20 17:05:42,194 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,46429,1689872718405 WAL splitting is done? wals=0, meta=false 2023-07-20 17:05:42,196 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ff374c4df33a87c27ad51fbe2c400e51, ASSIGN}, {pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN}] 2023-07-20 17:05:42,197 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=ff374c4df33a87c27ad51fbe2c400e51, ASSIGN 2023-07-20 17:05:42,197 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN 2023-07-20 17:05:42,198 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 17:05:42,198 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=ff374c4df33a87c27ad51fbe2c400e51, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 17:05:42,199 DEBUG [jenkins-hbase4:39439] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 17:05:42,199 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=ff374c4df33a87c27ad51fbe2c400e51, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:42,199 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=da051012dc01a99ec4c321e35c395da0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:42,200 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689872742199"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689872742199"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689872742199"}]},"ts":"1689872742199"} 2023-07-20 17:05:42,200 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689872721094.da051012dc01a99ec4c321e35c395da0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689872742199"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689872742199"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689872742199"}]},"ts":"1689872742199"} 2023-07-20 17:05:42,200 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:46429 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46429 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 17:05:42,202 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:46429 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46429 2023-07-20 17:05:42,289 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46429 this server is in the failed servers list 2023-07-20 17:05:42,303 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 17:05:42,305 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 17:05:42,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure da051012dc01a99ec4c321e35c395da0, server=jenkins-hbase4.apache.org,38295,1689872723718}] 2023-07-20 17:05:42,310 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE; OpenRegionProcedure ff374c4df33a87c27ad51fbe2c400e51, server=jenkins-hbase4.apache.org,38295,1689872723718}] 2023-07-20 17:05:42,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:42,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => da051012dc01a99ec4c321e35c395da0, NAME => 't,,1689872721094.da051012dc01a99ec4c321e35c395da0.', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:42,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:42,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689872721094.da051012dc01a99ec4c321e35c395da0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:42,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:42,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:42,468 INFO [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:42,469 DEBUG [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/f 2023-07-20 17:05:42,469 DEBUG [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/f 2023-07-20 17:05:42,470 INFO [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da051012dc01a99ec4c321e35c395da0 columnFamilyName f 2023-07-20 17:05:42,470 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:42,470 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ff374c4df33a87c27ad51fbe2c400e51, NAME => 'hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.', STARTKEY => '', ENDKEY => ''} 2023-07-20 17:05:42,471 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:42,471 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 17:05:42,471 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:42,471 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:42,474 INFO [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:42,476 DEBUG [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/info 2023-07-20 17:05:42,477 DEBUG [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/info 2023-07-20 17:05:42,477 INFO [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ff374c4df33a87c27ad51fbe2c400e51 columnFamilyName info 2023-07-20 17:05:42,478 INFO [StoreOpener-ff374c4df33a87c27ad51fbe2c400e51-1] regionserver.HStore(310): Store=ff374c4df33a87c27ad51fbe2c400e51/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:42,479 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:42,481 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:42,481 DEBUG [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] regionserver.HStore(539): loaded hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/f/40e2ab6e048b411c95b4e536195cea4d 2023-07-20 17:05:42,482 INFO [StoreOpener-da051012dc01a99ec4c321e35c395da0-1] regionserver.HStore(310): Store=da051012dc01a99ec4c321e35c395da0/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 17:05:42,482 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000005 2023-07-20 17:05:42,482 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:42,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:42,486 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000005 2023-07-20 17:05:42,486 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ff374c4df33a87c27ad51fbe2c400e51 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-20 17:05:42,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:42,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-20 17:05:42,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened da051012dc01a99ec4c321e35c395da0; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61268327, jitterRate=-0.08703078329563141}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 17:05:42,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for da051012dc01a99ec4c321e35c395da0: 2023-07-20 17:05:42,492 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689872721094.da051012dc01a99ec4c321e35c395da0., pid=30, masterSystemTime=1689872742461 2023-07-20 17:05:42,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:42,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:42,495 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=da051012dc01a99ec4c321e35c395da0, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:42,495 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689872721094.da051012dc01a99ec4c321e35c395da0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689872742495"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689872742495"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689872742495"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689872742495"}]},"ts":"1689872742495"} 2023-07-20 17:05:42,496 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46429 this server is in the failed servers list 2023-07-20 17:05:42,503 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-07-20 17:05:42,503 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure da051012dc01a99ec4c321e35c395da0, server=jenkins-hbase4.apache.org,38295,1689872723718 in 190 msec 2023-07-20 17:05:42,506 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=t, region=da051012dc01a99ec4c321e35c395da0, ASSIGN in 307 msec 2023-07-20 17:05:42,506 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/.tmp/info/0a06bdf6efb94a35b91370498f41bd06 2023-07-20 17:05:42,514 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/.tmp/info/0a06bdf6efb94a35b91370498f41bd06 as hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/info/0a06bdf6efb94a35b91370498f41bd06 2023-07-20 17:05:42,521 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/info/0a06bdf6efb94a35b91370498f41bd06, entries=2, sequenceid=5, filesize=4.8 K 2023-07-20 17:05:42,521 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for ff374c4df33a87c27ad51fbe2c400e51 in 35ms, sequenceid=5, compaction requested=false; wal=null 2023-07-20 17:05:42,522 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/0000000000000000005 2023-07-20 17:05:42,525 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:42,528 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-20 17:05:42,529 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ff374c4df33a87c27ad51fbe2c400e51; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61325717, jitterRate=-0.08617560565471649}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 17:05:42,529 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ff374c4df33a87c27ad51fbe2c400e51: 2023-07-20 17:05:42,530 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51., pid=31, masterSystemTime=1689872742461 2023-07-20 17:05:42,532 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:42,532 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:42,532 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=ff374c4df33a87c27ad51fbe2c400e51, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:42,533 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689872742532"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689872742532"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689872742532"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689872742532"}]},"ts":"1689872742532"} 2023-07-20 17:05:42,536 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=28 2023-07-20 17:05:42,537 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; OpenRegionProcedure ff374c4df33a87c27ad51fbe2c400e51, server=jenkins-hbase4.apache.org,38295,1689872723718 in 224 msec 2023-07-20 17:05:42,539 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=21 2023-07-20 17:05:42,539 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,46429,1689872718405 after splitting done 2023-07-20 17:05:42,539 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=ff374c4df33a87c27ad51fbe2c400e51, ASSIGN in 341 msec 2023-07-20 17:05:42,539 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,46429,1689872718405 from processing; numProcessing=0 2023-07-20 17:05:42,541 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,46429,1689872718405, splitWal=true, meta=true in 1.2640 sec 2023-07-20 17:05:42,806 DEBUG [Listener at localhost/41543] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x003d8b32 to 127.0.0.1:57299 2023-07-20 17:05:42,806 DEBUG [Listener at localhost/41543] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:42,806 INFO [Listener at localhost/41543] client.TestScannerTimeout(124): END ************ test2772 2023-07-20 17:05:42,820 INFO [Listener at localhost/41543] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=277 (was 296), OpenFileDescriptor=583 (was 573) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=175 (was 175), ProcessCount=171 (was 171), AvailableMemoryMB=5968 (was 6075) 2023-07-20 17:05:42,820 INFO [Listener at localhost/41543] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-20 17:05:42,820 INFO [Listener at localhost/41543] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-20 17:05:42,820 DEBUG [Listener at localhost/41543] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a84a831 to 127.0.0.1:57299 2023-07-20 17:05:42,821 DEBUG [Listener at localhost/41543] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:42,821 DEBUG [Listener at localhost/41543] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-20 17:05:42,821 DEBUG [Listener at localhost/41543] util.JVMClusterUtil(257): Found active master hash=925840766, stopped=false 2023-07-20 17:05:42,821 INFO [Listener at localhost/41543] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:42,824 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-20 17:05:42,824 INFO [Listener at localhost/41543] procedure2.ProcedureExecutor(629): Stopping 2023-07-20 17:05:42,824 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-20 17:05:42,824 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:42,824 DEBUG [Listener at localhost/41543] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06b95531 to 127.0.0.1:57299 2023-07-20 17:05:42,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 17:05:42,824 DEBUG [Listener at localhost/41543] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:42,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 17:05:42,825 INFO [Listener at localhost/41543] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,38295,1689872723718' ***** 2023-07-20 17:05:42,825 INFO [Listener at localhost/41543] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-20 17:05:42,825 INFO [RS:2;jenkins-hbase4:38295] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 17:05:42,825 INFO [RS:2;jenkins-hbase4:38295] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-20 17:05:42,825 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 17:05:42,825 INFO [RS:2;jenkins-hbase4:38295] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-20 17:05:42,825 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(3305): Received CLOSE for da051012dc01a99ec4c321e35c395da0 2023-07-20 17:05:42,825 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(3305): Received CLOSE for ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:42,826 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:42,826 DEBUG [RS:2;jenkins-hbase4:38295] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6cdca08e to 127.0.0.1:57299 2023-07-20 17:05:42,826 DEBUG [RS:2;jenkins-hbase4:38295] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:42,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing da051012dc01a99ec4c321e35c395da0, disabling compactions & flushes 2023-07-20 17:05:42,826 INFO [RS:2;jenkins-hbase4:38295] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 17:05:42,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:42,826 INFO [RS:2;jenkins-hbase4:38295] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 17:05:42,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:42,826 INFO [RS:2;jenkins-hbase4:38295] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 17:05:42,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689872721094.da051012dc01a99ec4c321e35c395da0. after waiting 0 ms 2023-07-20 17:05:42,826 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 17:05:42,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:42,827 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 17:05:42,827 DEBUG [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1478): Online Regions={da051012dc01a99ec4c321e35c395da0=t,,1689872721094.da051012dc01a99ec4c321e35c395da0., 1588230740=hbase:meta,,1.1588230740, ff374c4df33a87c27ad51fbe2c400e51=hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51.} 2023-07-20 17:05:42,827 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 17:05:42,827 DEBUG [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1504): Waiting on 1588230740, da051012dc01a99ec4c321e35c395da0, ff374c4df33a87c27ad51fbe2c400e51 2023-07-20 17:05:42,827 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 17:05:42,827 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 17:05:42,827 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 17:05:42,828 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 17:05:42,828 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-20 17:05:42,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/default/t/da051012dc01a99ec4c321e35c395da0/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-20 17:05:42,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:42,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for da051012dc01a99ec4c321e35c395da0: 2023-07-20 17:05:42,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689872721094.da051012dc01a99ec4c321e35c395da0. 2023-07-20 17:05:42,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ff374c4df33a87c27ad51fbe2c400e51, disabling compactions & flushes 2023-07-20 17:05:42,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:42,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:42,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. after waiting 0 ms 2023-07-20 17:05:42,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:42,841 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-20 17:05:42,841 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-20 17:05:42,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/namespace/ff374c4df33a87c27ad51fbe2c400e51/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-20 17:05:42,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:42,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ff374c4df33a87c27ad51fbe2c400e51: 2023-07-20 17:05:42,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689872720261.ff374c4df33a87c27ad51fbe2c400e51. 2023-07-20 17:05:42,855 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/.tmp/info/4b0eb27e542d4d80b8cb8fd58b4de438 2023-07-20 17:05:42,862 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/.tmp/info/4b0eb27e542d4d80b8cb8fd58b4de438 as hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info/4b0eb27e542d4d80b8cb8fd58b4de438 2023-07-20 17:05:42,869 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/info/4b0eb27e542d4d80b8cb8fd58b4de438, entries=16, sequenceid=24, filesize=6.4 K 2023-07-20 17:05:42,870 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 42ms, sequenceid=24, compaction requested=true 2023-07-20 17:05:42,877 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=17 2023-07-20 17:05:42,877 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 17:05:42,877 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 17:05:42,877 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 17:05:42,878 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 17:05:43,027 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,38295,1689872723718; all regions closed. 2023-07-20 17:05:43,033 DEBUG [RS:2;jenkins-hbase4:38295] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs 2023-07-20 17:05:43,034 INFO [RS:2;jenkins-hbase4:38295] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38295%2C1689872723718.meta:.meta(num 1689872741829) 2023-07-20 17:05:43,038 DEBUG [RS:2;jenkins-hbase4:38295] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/oldWALs 2023-07-20 17:05:43,038 INFO [RS:2;jenkins-hbase4:38295] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38295%2C1689872723718:(num 1689872723921) 2023-07-20 17:05:43,038 DEBUG [RS:2;jenkins-hbase4:38295] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:43,038 INFO [RS:2;jenkins-hbase4:38295] regionserver.LeaseManager(133): Closed leases 2023-07-20 17:05:43,038 INFO [RS:2;jenkins-hbase4:38295] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-20 17:05:43,039 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 17:05:43,039 INFO [RS:2;jenkins-hbase4:38295] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38295 2023-07-20 17:05:43,041 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:43,041 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38295,1689872723718 2023-07-20 17:05:43,041 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 17:05:43,042 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38295,1689872723718] 2023-07-20 17:05:43,042 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38295,1689872723718; numProcessing=1 2023-07-20 17:05:43,044 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38295,1689872723718 already deleted, retry=false 2023-07-20 17:05:43,045 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38295,1689872723718 expired; onlineServers=0 2023-07-20 17:05:43,045 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39439,1689872717206' ***** 2023-07-20 17:05:43,045 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-20 17:05:43,045 DEBUG [M:0;jenkins-hbase4:39439] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76123f45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 17:05:43,045 INFO [M:0;jenkins-hbase4:39439] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,39439,1689872717206 2023-07-20 17:05:43,045 INFO [M:0;jenkins-hbase4:39439] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,39439,1689872717206; all regions closed. 2023-07-20 17:05:43,045 DEBUG [M:0;jenkins-hbase4:39439] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 17:05:43,045 DEBUG [M:0;jenkins-hbase4:39439] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-20 17:05:43,045 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-20 17:05:43,045 DEBUG [M:0;jenkins-hbase4:39439] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-20 17:05:43,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689872719464] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689872719464,5,FailOnTimeoutGroup] 2023-07-20 17:05:43,046 INFO [M:0;jenkins-hbase4:39439] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-20 17:05:43,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689872719465] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689872719465,5,FailOnTimeoutGroup] 2023-07-20 17:05:43,046 INFO [M:0;jenkins-hbase4:39439] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-20 17:05:43,046 INFO [M:0;jenkins-hbase4:39439] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-20 17:05:43,046 DEBUG [M:0;jenkins-hbase4:39439] master.HMaster(1512): Stopping service threads 2023-07-20 17:05:43,046 INFO [M:0;jenkins-hbase4:39439] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-20 17:05:43,047 ERROR [M:0;jenkins-hbase4:39439] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (90527944) connection to localhost/127.0.0.1:36759 from jenkins,5,PEWorkerGroup] 2023-07-20 17:05:43,047 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-20 17:05:43,047 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 17:05:43,047 INFO [M:0;jenkins-hbase4:39439] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-20 17:05:43,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 17:05:43,047 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-20 17:05:43,048 DEBUG [M:0;jenkins-hbase4:39439] zookeeper.ZKUtil(398): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-20 17:05:43,048 WARN [M:0;jenkins-hbase4:39439] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-20 17:05:43,048 INFO [M:0;jenkins-hbase4:39439] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-20 17:05:43,048 INFO [M:0;jenkins-hbase4:39439] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-20 17:05:43,048 DEBUG [M:0;jenkins-hbase4:39439] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-20 17:05:43,049 INFO [M:0;jenkins-hbase4:39439] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 17:05:43,049 DEBUG [M:0;jenkins-hbase4:39439] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 17:05:43,049 DEBUG [M:0;jenkins-hbase4:39439] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-20 17:05:43,049 DEBUG [M:0;jenkins-hbase4:39439] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 17:05:43,049 INFO [M:0;jenkins-hbase4:39439] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=105.76 KB heapSize=127.45 KB 2023-07-20 17:05:43,065 INFO [M:0;jenkins-hbase4:39439] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=105.76 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f7481b6dd75a4e9483a2903d25a2073e 2023-07-20 17:05:43,073 DEBUG [M:0;jenkins-hbase4:39439] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f7481b6dd75a4e9483a2903d25a2073e as hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f7481b6dd75a4e9483a2903d25a2073e 2023-07-20 17:05:43,082 INFO [M:0;jenkins-hbase4:39439] regionserver.HStore(1080): Added hdfs://localhost:36759/user/jenkins/test-data/1146a7a4-d0b3-69d8-d358-ada5dc3a149f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f7481b6dd75a4e9483a2903d25a2073e, entries=31, sequenceid=244, filesize=8.4 K 2023-07-20 17:05:43,090 INFO [M:0;jenkins-hbase4:39439] regionserver.HRegion(2948): Finished flush of dataSize ~105.76 KB/108302, heapSize ~127.44 KB/130496, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 41ms, sequenceid=244, compaction requested=false 2023-07-20 17:05:43,093 INFO [M:0;jenkins-hbase4:39439] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 17:05:43,093 DEBUG [M:0;jenkins-hbase4:39439] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 17:05:43,105 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 17:05:43,105 INFO [M:0;jenkins-hbase4:39439] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-20 17:05:43,105 INFO [M:0;jenkins-hbase4:39439] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39439 2023-07-20 17:05:43,107 DEBUG [M:0;jenkins-hbase4:39439] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39439,1689872717206 already deleted, retry=false 2023-07-20 17:05:43,310 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 17:05:43,310 INFO [M:0;jenkins-hbase4:39439] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39439,1689872717206; zookeeper connection closed. 2023-07-20 17:05:43,310 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): master:39439-0x10183b1d5470000, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 17:05:43,410 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 17:05:43,410 INFO [RS:2;jenkins-hbase4:38295] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,38295,1689872723718; zookeeper connection closed. 2023-07-20 17:05:43,410 DEBUG [Listener at localhost/41543-EventThread] zookeeper.ZKWatcher(600): regionserver:38295-0x10183b1d5470008, quorum=127.0.0.1:57299, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 17:05:43,410 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@63d3f37f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@63d3f37f 2023-07-20 17:05:43,411 INFO [Listener at localhost/41543] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-20 17:05:43,411 WARN [Listener at localhost/41543] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-20 17:05:43,415 INFO [Listener at localhost/41543] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 17:05:43,423 WARN [BP-107507886-172.31.14.131-1689872714220 heartbeating to localhost/127.0.0.1:36759] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-20 17:05:43,423 WARN [BP-107507886-172.31.14.131-1689872714220 heartbeating to localhost/127.0.0.1:36759] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-107507886-172.31.14.131-1689872714220 (Datanode Uuid 5db548ba-844c-4201-b2b2-1954fb393813) service to localhost/127.0.0.1:36759 2023-07-20 17:05:43,425 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56/dfs/data/data3/current/BP-107507886-172.31.14.131-1689872714220] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 17:05:43,426 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56/dfs/data/data4/current/BP-107507886-172.31.14.131-1689872714220] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 17:05:43,428 WARN [Listener at localhost/41543] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-20 17:05:43,435 INFO [Listener at localhost/41543] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 17:05:43,440 WARN [BP-107507886-172.31.14.131-1689872714220 heartbeating to localhost/127.0.0.1:36759] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-20 17:05:43,441 WARN [BP-107507886-172.31.14.131-1689872714220 heartbeating to localhost/127.0.0.1:36759] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-107507886-172.31.14.131-1689872714220 (Datanode Uuid 6bdb3cad-7122-42c2-93fe-60972a5bafaa) service to localhost/127.0.0.1:36759 2023-07-20 17:05:43,441 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56/dfs/data/data1/current/BP-107507886-172.31.14.131-1689872714220] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 17:05:43,442 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f953e870-5cfe-5a06-301e-02924cbd7c67/cluster_b6e405fa-0ae4-6e73-1f8f-762ffdba1c56/dfs/data/data2/current/BP-107507886-172.31.14.131-1689872714220] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 17:05:43,478 INFO [Listener at localhost/41543] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 17:05:43,590 INFO [Listener at localhost/41543] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-20 17:05:43,627 INFO [Listener at localhost/41543] hbase.HBaseTestingUtility(1293): Minicluster is down