2023-07-18 07:05:13,189 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-18 07:05:13,426 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926 2023-07-18 07:05:13,443 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-18 07:05:13,444 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c, deleteOnExit=true 2023-07-18 07:05:13,444 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-18 07:05:13,445 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/test.cache.data in system properties and HBase conf 2023-07-18 07:05:13,445 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/hadoop.tmp.dir in system properties and HBase conf 2023-07-18 07:05:13,446 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/hadoop.log.dir in system properties and HBase conf 2023-07-18 07:05:13,447 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-18 07:05:13,447 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-18 07:05:13,447 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-18 07:05:13,557 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-18 07:05:13,931 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-18 07:05:13,935 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-18 07:05:13,935 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-18 07:05:13,935 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-18 07:05:13,936 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-18 07:05:13,936 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-18 07:05:13,936 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-18 07:05:13,937 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-18 07:05:13,937 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-18 07:05:13,938 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-18 07:05:13,938 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/nfs.dump.dir in system properties and HBase conf 2023-07-18 07:05:13,939 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/java.io.tmpdir in system properties and HBase conf 2023-07-18 07:05:13,939 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-18 07:05:13,939 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-18 07:05:13,940 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-18 07:05:14,459 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-18 07:05:14,464 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-18 07:05:14,741 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-18 07:05:14,901 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-18 07:05:14,921 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-18 07:05:14,963 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-18 07:05:14,994 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/java.io.tmpdir/Jetty_localhost_42593_hdfs____.5lmao5/webapp 2023-07-18 07:05:15,147 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42593 2023-07-18 07:05:15,158 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-18 07:05:15,158 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-18 07:05:15,536 WARN [Listener at localhost/37543] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-18 07:05:15,631 WARN [Listener at localhost/37543] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-18 07:05:15,650 WARN [Listener at localhost/37543] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-18 07:05:15,657 INFO [Listener at localhost/37543] log.Slf4jLog(67): jetty-6.1.26 2023-07-18 07:05:15,662 INFO [Listener at localhost/37543] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/java.io.tmpdir/Jetty_localhost_37017_datanode____ve0x21/webapp 2023-07-18 07:05:15,765 INFO [Listener at localhost/37543] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37017 2023-07-18 07:05:16,082 WARN [Listener at localhost/34933] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-18 07:05:16,095 WARN [Listener at localhost/34933] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-18 07:05:16,099 WARN [Listener at localhost/34933] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-18 07:05:16,102 INFO [Listener at localhost/34933] log.Slf4jLog(67): jetty-6.1.26 2023-07-18 07:05:16,109 INFO [Listener at localhost/34933] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/java.io.tmpdir/Jetty_localhost_35667_datanode____e02vcs/webapp 2023-07-18 07:05:16,219 INFO [Listener at localhost/34933] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35667 2023-07-18 07:05:16,234 WARN [Listener at localhost/37223] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-18 07:05:16,550 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x46ca6e780ddd5fe6: Processing first storage report for DS-1326d674-ade8-4b08-823f-06fe13d00ca3 from datanode a0460555-2eaf-4d85-9d1d-21d22e72bab6 2023-07-18 07:05:16,552 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x46ca6e780ddd5fe6: from storage DS-1326d674-ade8-4b08-823f-06fe13d00ca3 node DatanodeRegistration(127.0.0.1:46661, datanodeUuid=a0460555-2eaf-4d85-9d1d-21d22e72bab6, infoPort=35213, infoSecurePort=0, ipcPort=34933, storageInfo=lv=-57;cid=testClusterID;nsid=694251553;c=1689663914536), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-18 07:05:16,552 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x55ff362adcbce2ac: Processing first storage report for DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273 from datanode 5fb1d18f-cfa1-4cf8-adb2-38ba3992790e 2023-07-18 07:05:16,552 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x55ff362adcbce2ac: from storage DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273 node DatanodeRegistration(127.0.0.1:36191, datanodeUuid=5fb1d18f-cfa1-4cf8-adb2-38ba3992790e, infoPort=36791, infoSecurePort=0, ipcPort=37223, storageInfo=lv=-57;cid=testClusterID;nsid=694251553;c=1689663914536), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-18 07:05:16,552 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x46ca6e780ddd5fe6: Processing first storage report for DS-16eb73a5-0a19-4414-a4f1-8c298fd3a9f2 from datanode a0460555-2eaf-4d85-9d1d-21d22e72bab6 2023-07-18 07:05:16,553 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x46ca6e780ddd5fe6: from storage DS-16eb73a5-0a19-4414-a4f1-8c298fd3a9f2 node DatanodeRegistration(127.0.0.1:46661, datanodeUuid=a0460555-2eaf-4d85-9d1d-21d22e72bab6, infoPort=35213, infoSecurePort=0, ipcPort=34933, storageInfo=lv=-57;cid=testClusterID;nsid=694251553;c=1689663914536), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-18 07:05:16,553 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x55ff362adcbce2ac: Processing first storage report for DS-6b04f611-4e9a-4e44-82c6-d917c0f77df7 from datanode 5fb1d18f-cfa1-4cf8-adb2-38ba3992790e 2023-07-18 07:05:16,553 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x55ff362adcbce2ac: from storage DS-6b04f611-4e9a-4e44-82c6-d917c0f77df7 node DatanodeRegistration(127.0.0.1:36191, datanodeUuid=5fb1d18f-cfa1-4cf8-adb2-38ba3992790e, infoPort=36791, infoSecurePort=0, ipcPort=37223, storageInfo=lv=-57;cid=testClusterID;nsid=694251553;c=1689663914536), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-18 07:05:16,636 DEBUG [Listener at localhost/37223] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926 2023-07-18 07:05:16,698 INFO [Listener at localhost/37223] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c/zookeeper_0, clientPort=50929, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-18 07:05:16,714 INFO [Listener at localhost/37223] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50929 2023-07-18 07:05:16,723 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:16,725 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:17,385 INFO [Listener at localhost/37223] util.FSUtils(471): Created version file at hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9 with version=8 2023-07-18 07:05:17,385 INFO [Listener at localhost/37223] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/hbase-staging 2023-07-18 07:05:17,687 INFO [Listener at localhost/37223] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-18 07:05:18,140 INFO [Listener at localhost/37223] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-18 07:05:18,172 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:18,173 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:18,173 INFO [Listener at localhost/37223] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-18 07:05:18,173 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:18,173 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-18 07:05:18,315 INFO [Listener at localhost/37223] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-18 07:05:18,388 DEBUG [Listener at localhost/37223] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-18 07:05:18,486 INFO [Listener at localhost/37223] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46813 2023-07-18 07:05:18,496 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:18,498 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:18,518 INFO [Listener at localhost/37223] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46813 connecting to ZooKeeper ensemble=127.0.0.1:50929 2023-07-18 07:05:18,556 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:468130x0, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-18 07:05:18,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:46813-0x101773fcd8e0000 connected 2023-07-18 07:05:18,580 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-18 07:05:18,581 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 07:05:18,584 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-18 07:05:18,592 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46813 2023-07-18 07:05:18,592 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46813 2023-07-18 07:05:18,593 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46813 2023-07-18 07:05:18,594 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46813 2023-07-18 07:05:18,594 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46813 2023-07-18 07:05:18,600 INFO [Listener at localhost/37223] master.HMaster(444): hbase.rootdir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9, hbase.cluster.distributed=false 2023-07-18 07:05:18,671 INFO [Listener at localhost/37223] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-18 07:05:18,672 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:18,672 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:18,672 INFO [Listener at localhost/37223] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-18 07:05:18,672 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:18,672 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-18 07:05:18,678 INFO [Listener at localhost/37223] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-18 07:05:18,681 INFO [Listener at localhost/37223] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42631 2023-07-18 07:05:18,683 INFO [Listener at localhost/37223] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-18 07:05:18,690 DEBUG [Listener at localhost/37223] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-18 07:05:18,691 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:18,693 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:18,695 INFO [Listener at localhost/37223] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42631 connecting to ZooKeeper ensemble=127.0.0.1:50929 2023-07-18 07:05:18,700 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:426310x0, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-18 07:05:18,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:42631-0x101773fcd8e0001 connected 2023-07-18 07:05:18,701 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-18 07:05:18,702 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 07:05:18,703 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-18 07:05:18,704 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42631 2023-07-18 07:05:18,704 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42631 2023-07-18 07:05:18,704 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42631 2023-07-18 07:05:18,705 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42631 2023-07-18 07:05:18,705 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42631 2023-07-18 07:05:18,718 INFO [Listener at localhost/37223] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-18 07:05:18,719 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:18,719 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:18,720 INFO [Listener at localhost/37223] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-18 07:05:18,720 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:18,720 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-18 07:05:18,720 INFO [Listener at localhost/37223] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-18 07:05:18,722 INFO [Listener at localhost/37223] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43389 2023-07-18 07:05:18,722 INFO [Listener at localhost/37223] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-18 07:05:18,724 DEBUG [Listener at localhost/37223] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-18 07:05:18,725 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:18,727 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:18,728 INFO [Listener at localhost/37223] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43389 connecting to ZooKeeper ensemble=127.0.0.1:50929 2023-07-18 07:05:18,732 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:433890x0, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-18 07:05:18,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43389-0x101773fcd8e0002 connected 2023-07-18 07:05:18,734 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-18 07:05:18,735 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 07:05:18,735 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-18 07:05:18,736 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43389 2023-07-18 07:05:18,736 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43389 2023-07-18 07:05:18,736 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43389 2023-07-18 07:05:18,737 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43389 2023-07-18 07:05:18,737 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43389 2023-07-18 07:05:18,738 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:18,747 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-18 07:05:18,748 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:18,768 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-18 07:05:18,768 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-18 07:05:18,768 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:18,768 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-18 07:05:18,769 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-18 07:05:18,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46813,1689663917527 from backup master directory 2023-07-18 07:05:18,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-18 07:05:18,774 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:18,774 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-18 07:05:18,775 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-18 07:05:18,775 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:18,778 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-18 07:05:18,779 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-18 07:05:18,861 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/hbase.id with ID: 7ed3d068-ed92-4590-a281-cc5ca5fc66fb 2023-07-18 07:05:18,901 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:18,915 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:18,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x18a07576 to 127.0.0.1:50929 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 07:05:18,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2045b0c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 07:05:19,013 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-18 07:05:19,014 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-18 07:05:19,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-18 07:05:19,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-18 07:05:19,033 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-18 07:05:19,037 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-18 07:05:19,038 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 07:05:19,071 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/data/master/store-tmp 2023-07-18 07:05:19,104 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:19,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-18 07:05:19,105 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 07:05:19,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 07:05:19,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-18 07:05:19,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 07:05:19,105 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 07:05:19,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-18 07:05:19,107 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/WALs/jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:19,129 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46813%2C1689663917527, suffix=, logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/WALs/jenkins-hbase4.apache.org,46813,1689663917527, archiveDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/oldWALs, maxLogs=10 2023-07-18 07:05:19,184 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK] 2023-07-18 07:05:19,184 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK] 2023-07-18 07:05:19,192 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 07:05:19,254 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/WALs/jenkins-hbase4.apache.org,46813,1689663917527/jenkins-hbase4.apache.org%2C46813%2C1689663917527.1689663919139 2023-07-18 07:05:19,254 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK], DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK]] 2023-07-18 07:05:19,255 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:19,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:19,258 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-18 07:05:19,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-18 07:05:19,314 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-18 07:05:19,322 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-18 07:05:19,347 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-18 07:05:19,359 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:19,364 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-18 07:05:19,366 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-18 07:05:19,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-18 07:05:19,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-18 07:05:19,385 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61024729, jitterRate=-0.09066067636013031}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 07:05:19,386 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-18 07:05:19,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-18 07:05:19,406 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-18 07:05:19,406 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-18 07:05:19,408 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-18 07:05:19,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-18 07:05:19,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-18 07:05:19,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-18 07:05:19,467 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-18 07:05:19,472 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-18 07:05:19,497 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-18 07:05:19,501 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-18 07:05:19,502 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-18 07:05:19,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-18 07:05:19,511 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-18 07:05:19,514 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:19,515 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-18 07:05:19,515 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-18 07:05:19,527 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-18 07:05:19,532 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-18 07:05:19,532 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-18 07:05:19,532 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:19,532 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-18 07:05:19,533 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,46813,1689663917527, sessionid=0x101773fcd8e0000, setting cluster-up flag (Was=false) 2023-07-18 07:05:19,547 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:19,553 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-18 07:05:19,554 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:19,559 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:19,565 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-18 07:05:19,566 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:19,569 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.hbase-snapshot/.tmp 2023-07-18 07:05:19,641 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(951): ClusterId : 7ed3d068-ed92-4590-a281-cc5ca5fc66fb 2023-07-18 07:05:19,642 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(951): ClusterId : 7ed3d068-ed92-4590-a281-cc5ca5fc66fb 2023-07-18 07:05:19,646 DEBUG [RS:0;jenkins-hbase4:42631] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-18 07:05:19,646 DEBUG [RS:1;jenkins-hbase4:43389] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-18 07:05:19,653 DEBUG [RS:0;jenkins-hbase4:42631] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-18 07:05:19,653 DEBUG [RS:1;jenkins-hbase4:43389] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-18 07:05:19,654 DEBUG [RS:0;jenkins-hbase4:42631] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-18 07:05:19,654 DEBUG [RS:1;jenkins-hbase4:43389] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-18 07:05:19,657 DEBUG [RS:0;jenkins-hbase4:42631] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-18 07:05:19,657 DEBUG [RS:1;jenkins-hbase4:43389] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-18 07:05:19,659 DEBUG [RS:0;jenkins-hbase4:42631] zookeeper.ReadOnlyZKClient(139): Connect 0x1ad01384 to 127.0.0.1:50929 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 07:05:19,659 DEBUG [RS:1;jenkins-hbase4:43389] zookeeper.ReadOnlyZKClient(139): Connect 0x1bbbc2ce to 127.0.0.1:50929 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 07:05:19,673 DEBUG [RS:0;jenkins-hbase4:42631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1417feae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 07:05:19,673 DEBUG [RS:1;jenkins-hbase4:43389] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@157c3fc9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 07:05:19,673 DEBUG [RS:0;jenkins-hbase4:42631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@102878f4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-18 07:05:19,674 DEBUG [RS:1;jenkins-hbase4:43389] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1143a9be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-18 07:05:19,685 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-18 07:05:19,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-18 07:05:19,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-18 07:05:19,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-18 07:05:19,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-18 07:05:19,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-18 07:05:19,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-18 07:05:19,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,697 DEBUG [RS:0;jenkins-hbase4:42631] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:42631 2023-07-18 07:05:19,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689663949698 2023-07-18 07:05:19,698 DEBUG [RS:1;jenkins-hbase4:43389] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:43389 2023-07-18 07:05:19,701 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-18 07:05:19,702 INFO [RS:1;jenkins-hbase4:43389] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-18 07:05:19,702 INFO [RS:0;jenkins-hbase4:42631] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-18 07:05:19,703 INFO [RS:1;jenkins-hbase4:43389] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-18 07:05:19,703 INFO [RS:0;jenkins-hbase4:42631] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-18 07:05:19,703 DEBUG [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1022): About to register with Master. 2023-07-18 07:05:19,703 DEBUG [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1022): About to register with Master. 2023-07-18 07:05:19,705 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-18 07:05:19,706 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-18 07:05:19,706 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,46813,1689663917527 with isa=jenkins-hbase4.apache.org/172.31.14.131:42631, startcode=1689663918670 2023-07-18 07:05:19,706 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,46813,1689663917527 with isa=jenkins-hbase4.apache.org/172.31.14.131:43389, startcode=1689663918718 2023-07-18 07:05:19,710 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-18 07:05:19,712 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-18 07:05:19,720 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-18 07:05:19,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-18 07:05:19,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-18 07:05:19,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-18 07:05:19,722 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-18 07:05:19,724 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-18 07:05:19,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-18 07:05:19,727 DEBUG [RS:1;jenkins-hbase4:43389] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-18 07:05:19,727 DEBUG [RS:0;jenkins-hbase4:42631] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-18 07:05:19,727 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-18 07:05:19,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-18 07:05:19,729 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689663919729,5,FailOnTimeoutGroup] 2023-07-18 07:05:19,730 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689663919729,5,FailOnTimeoutGroup] 2023-07-18 07:05:19,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-18 07:05:19,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,732 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,754 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-18 07:05:19,756 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-18 07:05:19,757 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9 2023-07-18 07:05:19,781 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:19,785 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-18 07:05:19,788 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info 2023-07-18 07:05:19,789 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-18 07:05:19,790 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:19,791 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-18 07:05:19,794 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 07:05:19,795 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-18 07:05:19,796 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:19,796 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-18 07:05:19,798 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45899, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-18 07:05:19,798 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41911, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-18 07:05:19,799 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table 2023-07-18 07:05:19,799 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-18 07:05:19,800 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:19,802 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740 2023-07-18 07:05:19,803 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740 2023-07-18 07:05:19,807 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-18 07:05:19,809 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-18 07:05:19,812 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-18 07:05:19,813 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46813] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:19,814 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72783106, jitterRate=0.08455279469490051}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-18 07:05:19,814 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-18 07:05:19,814 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-18 07:05:19,814 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-18 07:05:19,814 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-18 07:05:19,814 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-18 07:05:19,814 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-18 07:05:19,818 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-18 07:05:19,818 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-18 07:05:19,824 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-18 07:05:19,824 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-18 07:05:19,824 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46813] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:19,832 DEBUG [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9 2023-07-18 07:05:19,832 DEBUG [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9 2023-07-18 07:05:19,832 DEBUG [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:37543 2023-07-18 07:05:19,832 DEBUG [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:37543 2023-07-18 07:05:19,832 DEBUG [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-18 07:05:19,832 DEBUG [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-18 07:05:19,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-18 07:05:19,841 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:19,842 DEBUG [RS:1;jenkins-hbase4:43389] zookeeper.ZKUtil(162): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:19,842 DEBUG [RS:0;jenkins-hbase4:42631] zookeeper.ZKUtil(162): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:19,842 WARN [RS:1;jenkins-hbase4:43389] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-18 07:05:19,843 WARN [RS:0;jenkins-hbase4:42631] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-18 07:05:19,843 INFO [RS:1;jenkins-hbase4:43389] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 07:05:19,843 INFO [RS:0;jenkins-hbase4:42631] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 07:05:19,844 DEBUG [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1948): logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:19,844 DEBUG [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1948): logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:19,844 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42631,1689663918670] 2023-07-18 07:05:19,844 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43389,1689663918718] 2023-07-18 07:05:19,852 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-18 07:05:19,855 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-18 07:05:19,855 DEBUG [RS:0;jenkins-hbase4:42631] zookeeper.ZKUtil(162): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:19,855 DEBUG [RS:1;jenkins-hbase4:43389] zookeeper.ZKUtil(162): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:19,855 DEBUG [RS:0;jenkins-hbase4:42631] zookeeper.ZKUtil(162): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:19,856 DEBUG [RS:1;jenkins-hbase4:43389] zookeeper.ZKUtil(162): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:19,866 DEBUG [RS:1;jenkins-hbase4:43389] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-18 07:05:19,866 DEBUG [RS:0;jenkins-hbase4:42631] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-18 07:05:19,876 INFO [RS:1;jenkins-hbase4:43389] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-18 07:05:19,876 INFO [RS:0;jenkins-hbase4:42631] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-18 07:05:19,897 INFO [RS:1;jenkins-hbase4:43389] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-18 07:05:19,897 INFO [RS:0;jenkins-hbase4:42631] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-18 07:05:19,902 INFO [RS:1;jenkins-hbase4:43389] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-18 07:05:19,902 INFO [RS:0;jenkins-hbase4:42631] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-18 07:05:19,902 INFO [RS:1;jenkins-hbase4:43389] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,902 INFO [RS:0;jenkins-hbase4:42631] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,903 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-18 07:05:19,903 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-18 07:05:19,911 INFO [RS:0;jenkins-hbase4:42631] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,911 INFO [RS:1;jenkins-hbase4:43389] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,911 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,911 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,912 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,912 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,912 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,912 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,912 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,912 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,912 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,912 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,912 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-18 07:05:19,912 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-18 07:05:19,913 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,913 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,913 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,913 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,913 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,913 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,913 DEBUG [RS:0;jenkins-hbase4:42631] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,913 DEBUG [RS:1;jenkins-hbase4:43389] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:19,914 INFO [RS:1;jenkins-hbase4:43389] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,915 INFO [RS:0;jenkins-hbase4:42631] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,915 INFO [RS:1;jenkins-hbase4:43389] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,915 INFO [RS:0;jenkins-hbase4:42631] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,915 INFO [RS:1;jenkins-hbase4:43389] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,915 INFO [RS:0;jenkins-hbase4:42631] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,931 INFO [RS:0;jenkins-hbase4:42631] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-18 07:05:19,931 INFO [RS:1;jenkins-hbase4:43389] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-18 07:05:19,934 INFO [RS:1;jenkins-hbase4:43389] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43389,1689663918718-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,934 INFO [RS:0;jenkins-hbase4:42631] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42631,1689663918670-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:19,950 INFO [RS:1;jenkins-hbase4:43389] regionserver.Replication(203): jenkins-hbase4.apache.org,43389,1689663918718 started 2023-07-18 07:05:19,951 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,43389,1689663918718, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43389, sessionid=0x101773fcd8e0002 2023-07-18 07:05:19,951 DEBUG [RS:1;jenkins-hbase4:43389] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-18 07:05:19,951 DEBUG [RS:1;jenkins-hbase4:43389] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:19,951 DEBUG [RS:1;jenkins-hbase4:43389] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43389,1689663918718' 2023-07-18 07:05:19,951 DEBUG [RS:1;jenkins-hbase4:43389] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-18 07:05:19,952 DEBUG [RS:1;jenkins-hbase4:43389] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-18 07:05:19,952 DEBUG [RS:1;jenkins-hbase4:43389] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-18 07:05:19,952 DEBUG [RS:1;jenkins-hbase4:43389] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-18 07:05:19,953 DEBUG [RS:1;jenkins-hbase4:43389] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:19,953 DEBUG [RS:1;jenkins-hbase4:43389] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43389,1689663918718' 2023-07-18 07:05:19,953 DEBUG [RS:1;jenkins-hbase4:43389] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-18 07:05:19,953 DEBUG [RS:1;jenkins-hbase4:43389] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-18 07:05:19,954 DEBUG [RS:1;jenkins-hbase4:43389] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-18 07:05:19,954 INFO [RS:1;jenkins-hbase4:43389] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-18 07:05:19,954 INFO [RS:1;jenkins-hbase4:43389] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-18 07:05:19,954 INFO [RS:0;jenkins-hbase4:42631] regionserver.Replication(203): jenkins-hbase4.apache.org,42631,1689663918670 started 2023-07-18 07:05:19,954 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,42631,1689663918670, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42631, sessionid=0x101773fcd8e0001 2023-07-18 07:05:19,954 DEBUG [RS:0;jenkins-hbase4:42631] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-18 07:05:19,954 DEBUG [RS:0;jenkins-hbase4:42631] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:19,955 DEBUG [RS:0;jenkins-hbase4:42631] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42631,1689663918670' 2023-07-18 07:05:19,955 DEBUG [RS:0;jenkins-hbase4:42631] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-18 07:05:19,956 DEBUG [RS:0;jenkins-hbase4:42631] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-18 07:05:19,957 DEBUG [RS:0;jenkins-hbase4:42631] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-18 07:05:19,957 DEBUG [RS:0;jenkins-hbase4:42631] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-18 07:05:19,957 DEBUG [RS:0;jenkins-hbase4:42631] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:19,957 DEBUG [RS:0;jenkins-hbase4:42631] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42631,1689663918670' 2023-07-18 07:05:19,957 DEBUG [RS:0;jenkins-hbase4:42631] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-18 07:05:19,958 DEBUG [RS:0;jenkins-hbase4:42631] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-18 07:05:19,958 DEBUG [RS:0;jenkins-hbase4:42631] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-18 07:05:19,958 INFO [RS:0;jenkins-hbase4:42631] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-18 07:05:19,959 INFO [RS:0;jenkins-hbase4:42631] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-18 07:05:20,007 DEBUG [jenkins-hbase4:46813] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-18 07:05:20,010 DEBUG [jenkins-hbase4:46813] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-18 07:05:20,016 DEBUG [jenkins-hbase4:46813] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 07:05:20,016 DEBUG [jenkins-hbase4:46813] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-18 07:05:20,016 DEBUG [jenkins-hbase4:46813] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-18 07:05:20,019 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42631,1689663918670, state=OPENING 2023-07-18 07:05:20,026 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-18 07:05:20,028 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:20,029 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 07:05:20,034 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42631,1689663918670}] 2023-07-18 07:05:20,064 INFO [RS:0;jenkins-hbase4:42631] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42631%2C1689663918670, suffix=, logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670, archiveDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs, maxLogs=32 2023-07-18 07:05:20,064 INFO [RS:1;jenkins-hbase4:43389] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43389%2C1689663918718, suffix=, logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718, archiveDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs, maxLogs=32 2023-07-18 07:05:20,087 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK] 2023-07-18 07:05:20,087 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK] 2023-07-18 07:05:20,087 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK] 2023-07-18 07:05:20,088 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK] 2023-07-18 07:05:20,096 INFO [RS:1;jenkins-hbase4:43389] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068 2023-07-18 07:05:20,097 DEBUG [RS:1;jenkins-hbase4:43389] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK], DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK]] 2023-07-18 07:05:20,101 INFO [RS:0;jenkins-hbase4:42631] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068 2023-07-18 07:05:20,101 DEBUG [RS:0;jenkins-hbase4:42631] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK], DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK]] 2023-07-18 07:05:20,219 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:20,223 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-18 07:05:20,226 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-18 07:05:20,237 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-18 07:05:20,238 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 07:05:20,241 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta, suffix=.meta, logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670, archiveDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs, maxLogs=32 2023-07-18 07:05:20,259 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK] 2023-07-18 07:05:20,259 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK] 2023-07-18 07:05:20,266 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta 2023-07-18 07:05:20,267 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK], DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK]] 2023-07-18 07:05:20,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:20,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-18 07:05:20,295 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-18 07:05:20,318 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-18 07:05:20,325 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-18 07:05:20,325 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:20,326 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-18 07:05:20,326 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-18 07:05:20,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-18 07:05:20,331 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info 2023-07-18 07:05:20,331 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info 2023-07-18 07:05:20,331 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-18 07:05:20,332 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:20,332 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-18 07:05:20,334 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 07:05:20,334 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 07:05:20,335 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-18 07:05:20,335 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:20,336 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-18 07:05:20,337 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table 2023-07-18 07:05:20,337 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table 2023-07-18 07:05:20,338 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-18 07:05:20,338 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:20,340 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740 2023-07-18 07:05:20,343 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740 2023-07-18 07:05:20,346 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-18 07:05:20,349 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-18 07:05:20,350 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65885951, jitterRate=-0.01822282373905182}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-18 07:05:20,350 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-18 07:05:20,361 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689663920211 2023-07-18 07:05:20,378 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-18 07:05:20,379 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-18 07:05:20,379 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42631,1689663918670, state=OPEN 2023-07-18 07:05:20,382 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 07:05:20,382 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 07:05:20,386 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-18 07:05:20,386 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42631,1689663918670 in 348 msec 2023-07-18 07:05:20,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-18 07:05:20,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 551 msec 2023-07-18 07:05:20,395 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 782 msec 2023-07-18 07:05:20,396 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689663920396, completionTime=-1 2023-07-18 07:05:20,396 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-18 07:05:20,396 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-18 07:05:20,465 DEBUG [hconnection-0x450610f5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:20,469 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52992, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:20,486 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-18 07:05:20,486 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689663980486 2023-07-18 07:05:20,486 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689664040486 2023-07-18 07:05:20,486 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 89 msec 2023-07-18 07:05:20,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46813,1689663917527-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:20,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46813,1689663917527-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:20,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46813,1689663917527-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:20,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46813, period=300000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:20,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:20,516 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-18 07:05:20,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-18 07:05:20,526 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-18 07:05:20,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-18 07:05:20,538 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-18 07:05:20,541 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-18 07:05:20,564 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:20,566 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09 empty. 2023-07-18 07:05:20,567 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:20,567 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-18 07:05:20,595 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-18 07:05:20,597 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => a53d19b08d23f40ffad9dc6f21a56d09, NAME => 'hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp 2023-07-18 07:05:20,614 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:20,615 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing a53d19b08d23f40ffad9dc6f21a56d09, disabling compactions & flushes 2023-07-18 07:05:20,615 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:20,615 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:20,615 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. after waiting 0 ms 2023-07-18 07:05:20,615 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:20,615 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:20,615 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for a53d19b08d23f40ffad9dc6f21a56d09: 2023-07-18 07:05:20,619 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-18 07:05:20,634 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689663920622"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689663920622"}]},"ts":"1689663920622"} 2023-07-18 07:05:20,664 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-18 07:05:20,666 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-18 07:05:20,671 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689663920666"}]},"ts":"1689663920666"} 2023-07-18 07:05:20,675 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-18 07:05:20,680 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-18 07:05:20,681 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 07:05:20,682 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-18 07:05:20,682 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-18 07:05:20,684 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a53d19b08d23f40ffad9dc6f21a56d09, ASSIGN}] 2023-07-18 07:05:20,687 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a53d19b08d23f40ffad9dc6f21a56d09, ASSIGN 2023-07-18 07:05:20,689 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a53d19b08d23f40ffad9dc6f21a56d09, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43389,1689663918718; forceNewPlan=false, retain=false 2023-07-18 07:05:20,842 INFO [jenkins-hbase4:46813] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-18 07:05:20,842 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a53d19b08d23f40ffad9dc6f21a56d09, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:20,843 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689663920842"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689663920842"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689663920842"}]},"ts":"1689663920842"} 2023-07-18 07:05:20,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure a53d19b08d23f40ffad9dc6f21a56d09, server=jenkins-hbase4.apache.org,43389,1689663918718}] 2023-07-18 07:05:21,001 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:21,001 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-18 07:05:21,004 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41468, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-18 07:05:21,009 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:21,010 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a53d19b08d23f40ffad9dc6f21a56d09, NAME => 'hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:21,011 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:21,011 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:21,011 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:21,011 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:21,013 INFO [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:21,015 DEBUG [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/info 2023-07-18 07:05:21,015 DEBUG [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/info 2023-07-18 07:05:21,016 INFO [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a53d19b08d23f40ffad9dc6f21a56d09 columnFamilyName info 2023-07-18 07:05:21,017 INFO [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] regionserver.HStore(310): Store=a53d19b08d23f40ffad9dc6f21a56d09/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:21,018 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:21,019 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:21,024 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:21,055 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-18 07:05:21,058 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a53d19b08d23f40ffad9dc6f21a56d09; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64112826, jitterRate=-0.044644445180892944}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 07:05:21,058 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a53d19b08d23f40ffad9dc6f21a56d09: 2023-07-18 07:05:21,062 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09., pid=6, masterSystemTime=1689663921000 2023-07-18 07:05:21,067 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:21,068 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:21,069 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a53d19b08d23f40ffad9dc6f21a56d09, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:21,070 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689663921069"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689663921069"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689663921069"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689663921069"}]},"ts":"1689663921069"} 2023-07-18 07:05:21,078 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-18 07:05:21,078 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure a53d19b08d23f40ffad9dc6f21a56d09, server=jenkins-hbase4.apache.org,43389,1689663918718 in 227 msec 2023-07-18 07:05:21,081 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-18 07:05:21,081 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a53d19b08d23f40ffad9dc6f21a56d09, ASSIGN in 394 msec 2023-07-18 07:05:21,083 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-18 07:05:21,083 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689663921083"}]},"ts":"1689663921083"} 2023-07-18 07:05:21,086 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-18 07:05:21,089 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-18 07:05:21,092 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 563 msec 2023-07-18 07:05:21,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-18 07:05:21,158 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-18 07:05:21,159 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:21,182 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:21,184 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41478, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:21,197 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-18 07:05:21,217 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-18 07:05:21,223 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-07-18 07:05:21,234 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-18 07:05:21,244 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-18 07:05:21,248 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-07-18 07:05:21,260 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-18 07:05:21,263 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-18 07:05:21,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.489sec 2023-07-18 07:05:21,267 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-18 07:05:21,269 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-18 07:05:21,269 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-18 07:05:21,271 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46813,1689663917527-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-18 07:05:21,271 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46813,1689663917527-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-18 07:05:21,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-18 07:05:21,361 DEBUG [Listener at localhost/37223] zookeeper.ReadOnlyZKClient(139): Connect 0x30d0fb75 to 127.0.0.1:50929 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 07:05:21,367 DEBUG [Listener at localhost/37223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35f511f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 07:05:21,381 DEBUG [hconnection-0x5ab85974-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:21,391 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52998, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:21,401 INFO [Listener at localhost/37223] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:21,410 DEBUG [Listener at localhost/37223] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-18 07:05:21,413 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41870, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-18 07:05:21,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-18 07:05:21,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-18 07:05:21,430 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-18 07:05:21,432 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-18 07:05:21,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-18 07:05:21,435 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:21,436 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588 empty. 2023-07-18 07:05:21,438 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:21,438 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-18 07:05:21,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-18 07:05:21,457 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-18 07:05:21,459 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => f8db4d5f36fb837f8dd2c3c681e31588, NAME => 't,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/.tmp 2023-07-18 07:05:21,477 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:21,477 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing f8db4d5f36fb837f8dd2c3c681e31588, disabling compactions & flushes 2023-07-18 07:05:21,477 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:21,477 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:21,477 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. after waiting 0 ms 2023-07-18 07:05:21,477 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:21,477 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:21,477 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for f8db4d5f36fb837f8dd2c3c681e31588: 2023-07-18 07:05:21,481 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-18 07:05:21,483 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689663921482"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689663921482"}]},"ts":"1689663921482"} 2023-07-18 07:05:21,485 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-18 07:05:21,486 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-18 07:05:21,487 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689663921487"}]},"ts":"1689663921487"} 2023-07-18 07:05:21,489 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-18 07:05:21,492 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-18 07:05:21,493 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 07:05:21,494 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-18 07:05:21,494 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-18 07:05:21,494 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN}] 2023-07-18 07:05:21,496 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN 2023-07-18 07:05:21,497 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42631,1689663918670; forceNewPlan=false, retain=false 2023-07-18 07:05:21,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-18 07:05:21,647 INFO [jenkins-hbase4:46813] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-18 07:05:21,648 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f8db4d5f36fb837f8dd2c3c681e31588, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:21,649 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689663921648"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689663921648"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689663921648"}]},"ts":"1689663921648"} 2023-07-18 07:05:21,652 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f8db4d5f36fb837f8dd2c3c681e31588, server=jenkins-hbase4.apache.org,42631,1689663918670}] 2023-07-18 07:05:21,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-18 07:05:21,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:21,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f8db4d5f36fb837f8dd2c3c681e31588, NAME => 't,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:21,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:21,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:21,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:21,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:21,813 INFO [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:21,816 DEBUG [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/f 2023-07-18 07:05:21,816 DEBUG [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/f 2023-07-18 07:05:21,816 INFO [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f8db4d5f36fb837f8dd2c3c681e31588 columnFamilyName f 2023-07-18 07:05:21,817 INFO [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] regionserver.HStore(310): Store=f8db4d5f36fb837f8dd2c3c681e31588/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:21,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:21,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:21,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:21,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-18 07:05:21,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f8db4d5f36fb837f8dd2c3c681e31588; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67439028, jitterRate=0.004919826984405518}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 07:05:21,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f8db4d5f36fb837f8dd2c3c681e31588: 2023-07-18 07:05:21,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588., pid=11, masterSystemTime=1689663921805 2023-07-18 07:05:21,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:21,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:21,833 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f8db4d5f36fb837f8dd2c3c681e31588, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:21,833 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689663921833"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689663921833"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689663921833"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689663921833"}]},"ts":"1689663921833"} 2023-07-18 07:05:21,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-18 07:05:21,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f8db4d5f36fb837f8dd2c3c681e31588, server=jenkins-hbase4.apache.org,42631,1689663918670 in 184 msec 2023-07-18 07:05:21,843 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-18 07:05:21,843 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN in 345 msec 2023-07-18 07:05:21,844 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-18 07:05:21,844 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689663921844"}]},"ts":"1689663921844"} 2023-07-18 07:05:21,846 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-18 07:05:21,850 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-18 07:05:21,852 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 425 msec 2023-07-18 07:05:22,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-18 07:05:22,053 INFO [Listener at localhost/37223] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-18 07:05:22,053 DEBUG [Listener at localhost/37223] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-18 07:05:22,055 INFO [Listener at localhost/37223] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-18 07:05:22,059 INFO [Listener at localhost/37223] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-18 07:05:22,060 INFO [Listener at localhost/37223] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-18 07:05:22,060 INFO [Listener at localhost/37223] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-18 07:05:22,107 INFO [Listener at localhost/37223] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=130, ProcessCount=169, AvailableMemoryMB=6149 2023-07-18 07:05:22,109 INFO [Listener at localhost/37223] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-18 07:05:22,113 DEBUG [Listener at localhost/37223] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-18 07:05:22,114 DEBUG [Listener at localhost/37223] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:22,114 INFO [Listener at localhost/37223] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-18 07:05:22,114 INFO [Listener at localhost/37223] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-18 07:05:22,118 INFO [Listener at localhost/37223] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689663921083/Put/vlen=2/seqid=0} 2023-07-18 07:05:22,118 INFO [Listener at localhost/37223] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-18 07:05:22,119 INFO [Listener at localhost/37223] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09./info:regioninfo/1689663921069/Put/vlen=41/seqid=0, hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09./info:seqnumDuringOpen/1689663921069/Put/vlen=8/seqid=0, hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09./info:server/1689663921069/Put/vlen=31/seqid=0, hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09./info:serverstartcode/1689663921069/Put/vlen=8/seqid=0, hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09./info:sn/1689663920842/Put/vlen=45/seqid=0, hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09./info:state/1689663921069/Put/vlen=4/seqid=0} 2023-07-18 07:05:22,119 INFO [Listener at localhost/37223] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => a53d19b08d23f40ffad9dc6f21a56d09, NAME => 'hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:22,119 INFO [Listener at localhost/37223] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689663921844/Put/vlen=2/seqid=0} 2023-07-18 07:05:22,119 INFO [Listener at localhost/37223] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-18 07:05:22,119 INFO [Listener at localhost/37223] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588./info:regioninfo/1689663921833/Put/vlen=35/seqid=0, t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588./info:seqnumDuringOpen/1689663921833/Put/vlen=8/seqid=0, t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588./info:server/1689663921833/Put/vlen=31/seqid=0, t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588./info:serverstartcode/1689663921833/Put/vlen=8/seqid=0, t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588./info:sn/1689663921648/Put/vlen=45/seqid=0, t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588./info:state/1689663921833/Put/vlen=4/seqid=0} 2023-07-18 07:05:22,120 INFO [Listener at localhost/37223] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => f8db4d5f36fb837f8dd2c3c681e31588, NAME => 't,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:22,121 DEBUG [Listener at localhost/37223] zookeeper.ReadOnlyZKClient(139): Connect 0x243942ca to 127.0.0.1:50929 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 07:05:22,126 DEBUG [Listener at localhost/37223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cf43c4d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 07:05:22,127 INFO [Listener at localhost/37223] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-18 07:05:22,127 INFO [Listener at localhost/37223] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-18 07:05:22,129 DEBUG [hconnection-0xc8a45ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:22,132 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53008, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:22,139 INFO [Listener at localhost/37223] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-18 07:05:22,142 ERROR [Listener at localhost/37223] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,42631,1689663918670: die! ***** 2023-07-18 07:05:22,143 ERROR [Listener at localhost/37223] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-18 07:05:22,146 DEBUG [Listener at localhost/37223] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-18 07:05:22,149 DEBUG [Listener at localhost/37223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-18 07:05:22,149 DEBUG [Listener at localhost/37223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-18 07:05:22,150 DEBUG [Listener at localhost/37223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-18 07:05:22,150 INFO [Listener at localhost/37223] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 963641344, "init": 513802240, "max": 2051014656, "used": 183088072 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 110992960 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-18 07:05:22,154 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46813] master.MasterRpcServices(612): jenkins-hbase4.apache.org,42631,1689663918670 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,42631,1689663918670: die! ***** 2023-07-18 07:05:22,156 INFO [Listener at localhost/37223] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42631,1689663918670' ***** 2023-07-18 07:05:22,157 INFO [Listener at localhost/37223] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-18 07:05:22,157 INFO [RS:0;jenkins-hbase4:42631] regionserver.HeapMemoryManager(220): Stopping 2023-07-18 07:05:22,157 INFO [RS:0;jenkins-hbase4:42631] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-18 07:05:22,157 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-18 07:05:22,158 INFO [RS:0;jenkins-hbase4:42631] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-18 07:05:22,158 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(3305): Received CLOSE for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:22,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42631] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 7730618572166135819 2023-07-18 07:05:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42631] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42631,1689663918670 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 07:05:22,159 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:22,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f8db4d5f36fb837f8dd2c3c681e31588, disabling compactions & flushes 2023-07-18 07:05:22,160 DEBUG [RS:0;jenkins-hbase4:42631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ad01384 to 127.0.0.1:50929 2023-07-18 07:05:22,160 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:22,160 DEBUG [RS:0;jenkins-hbase4:42631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:22,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42631] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:53008 deadline: 1689663982158, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42631,1689663918670 aborting 2023-07-18 07:05:22,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:22,160 INFO [RS:0;jenkins-hbase4:42631] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-18 07:05:22,160 INFO [RS:0;jenkins-hbase4:42631] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-18 07:05:22,160 INFO [RS:0;jenkins-hbase4:42631] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-18 07:05:22,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. after waiting 0 ms 2023-07-18 07:05:22,160 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-18 07:05:22,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:22,161 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-18 07:05:22,161 DEBUG [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, f8db4d5f36fb837f8dd2c3c681e31588=t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.} 2023-07-18 07:05:22,161 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-18 07:05:22,162 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-18 07:05:22,162 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:22,162 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-18 07:05:22,162 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-18 07:05:22,162 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-18 07:05:22,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:22,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f8db4d5f36fb837f8dd2c3c681e31588: 2023-07-18 07:05:22,163 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:22,163 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-18 07:05:22,163 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-18 07:05:22,164 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-18 07:05:22,164 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-18 07:05:22,164 DEBUG [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-18 07:05:22,164 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-18 07:05:22,179 DEBUG [Listener at localhost/37223] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42631,1689663918670 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42631,1689663918670 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-18 07:05:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42631] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42631,1689663918670 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 07:05:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42631] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:53008 deadline: 1689663982182, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42631,1689663918670 aborting 2023-07-18 07:05:22,286 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42631] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42631,1689663918670 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 07:05:22,287 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42631] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:53008 deadline: 1689663982286, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42631,1689663918670 aborting 2023-07-18 07:05:22,364 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,42631,1689663918670; all regions closed. 2023-07-18 07:05:22,379 DEBUG [RS:0;jenkins-hbase4:42631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:22,380 INFO [RS:0;jenkins-hbase4:42631] regionserver.LeaseManager(133): Closed leases 2023-07-18 07:05:22,381 INFO [RS:0;jenkins-hbase4:42631] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-18 07:05:22,381 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-18 07:05:22,382 INFO [RS:0;jenkins-hbase4:42631] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42631 2023-07-18 07:05:22,391 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:22,391 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42631 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42631 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 07:05:22,391 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:22,391 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:22,391 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:22,392 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:22,392 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42631 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42631 2023-07-18 07:05:22,393 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42631,1689663918670] 2023-07-18 07:05:22,394 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42631,1689663918670; numProcessing=1 2023-07-18 07:05:22,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:22,395 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,42631,1689663918670 znode expired, triggering replicatorRemoved event 2023-07-18 07:05:22,396 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42631,1689663918670 already deleted, retry=false 2023-07-18 07:05:22,396 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,42631,1689663918670 on jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:22,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:22,404 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,42631,1689663918670, splitWal=true, meta=true 2023-07-18 07:05:22,404 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,42631,1689663918670 (carryingMeta=true) jenkins-hbase4.apache.org,42631,1689663918670/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@76ffc104[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-18 07:05:22,407 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42631,1689663918670, splitWal=true, meta=true 2023-07-18 07:05:22,408 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42631,1689663918670, splitWal=true, meta=true, isMeta: true 2023-07-18 07:05:22,410 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting 2023-07-18 07:05:22,412 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,42631,1689663918670 WAL count=1, meta=true 2023-07-18 07:05:22,416 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta}] 2023-07-18 07:05:22,424 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:22,425 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta, worker=jenkins-hbase4.apache.org,43389,1689663918718}] 2023-07-18 07:05:22,493 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 07:05:22,494 INFO [RS:0;jenkins-hbase4:42631] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42631,1689663918670; zookeeper connection closed. 2023-07-18 07:05:22,494 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:42631-0x101773fcd8e0001, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 07:05:22,494 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6376f112] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6376f112 2023-07-18 07:05:22,589 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43389] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-18 07:05:22,594 DEBUG [hconnection-0xc8a45ae-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42631 this server is in the failed servers list 2023-07-18 07:05:22,612 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta, size=3.1 K (3185bytes) 2023-07-18 07:05:22,613 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta 2023-07-18 07:05:22,618 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta after 5ms 2023-07-18 07:05:22,623 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta took 13ms 2023-07-18 07:05:22,638 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-18 07:05:22,640 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta so closing down 2023-07-18 07:05:22,640 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-18 07:05:22,646 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-18 07:05:22,649 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta.temp 2023-07-18 07:05:22,651 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-18 07:05:22,663 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-18 07:05:22,665 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta.temp to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-18 07:05:22,666 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 42 ms; skipped=1; WAL=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-18 07:05:22,666 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta, journal: Splitting hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta, size=3.1 K (3185bytes) at 1689663922612Finishing writing output for hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta so closing down at 1689663922640 (+28 ms)Creating recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta.temp at 1689663922649 (+9 ms)3 split writer threads finished at 1689663922651 (+2 ms)Closed recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689663922663 (+12 ms)Rename recovered edits hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta.temp to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689663922665 (+2 ms)Processed 11 edits across 1 Regions in 42 ms; skipped=1; WAL=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689663922666 (+1 ms) 2023-07-18 07:05:22,667 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta 2023-07-18 07:05:22,669 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-18 07:05:22,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.HMaster(3641): Remote procedure done, pid=14 2023-07-18 07:05:22,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] wal.WALSplitUtil(141): Moved hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs 2023-07-18 07:05:22,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-18 07:05:22,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta, worker=jenkins-hbase4.apache.org,43389,1689663918718 in 257 msec 2023-07-18 07:05:22,687 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:22,691 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-18 07:05:22,691 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C42631%2C1689663918670.meta.1689663920243.meta, worker=jenkins-hbase4.apache.org,43389,1689663918718 in 273 msec 2023-07-18 07:05:22,694 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting dir is empty, no logs to split. 2023-07-18 07:05:22,694 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,42631,1689663918670 WAL count=0, meta=true 2023-07-18 07:05:22,694 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42631,1689663918670 WAL splitting is done? wals=0, meta=true 2023-07-18 07:05:22,696 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-18 07:05:22,697 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-18 07:05:22,698 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 07:05:22,848 DEBUG [jenkins-hbase4:46813] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-18 07:05:22,849 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43389,1689663918718, state=OPENING 2023-07-18 07:05:22,850 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 07:05:22,851 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 07:05:22,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43389,1689663918718}] 2023-07-18 07:05:22,896 WARN [ReadOnlyZKClient-127.0.0.1:50929@0x243942ca] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-18 07:05:22,896 DEBUG [hconnection-0xc8a45ae-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:22,900 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:22,900 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43389] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:41482 deadline: 1689663982900, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:22,916 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-18 07:05:23,009 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-18 07:05:23,009 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 07:05:23,011 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta, suffix=.meta, logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718, archiveDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs, maxLogs=32 2023-07-18 07:05:23,029 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK] 2023-07-18 07:05:23,030 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK] 2023-07-18 07:05:23,034 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta 2023-07-18 07:05:23,034 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK], DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK]] 2023-07-18 07:05:23,034 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:23,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-18 07:05:23,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-18 07:05:23,035 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-18 07:05:23,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-18 07:05:23,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:23,036 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-18 07:05:23,036 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-18 07:05:23,037 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-18 07:05:23,039 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info 2023-07-18 07:05:23,039 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info 2023-07-18 07:05:23,039 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-18 07:05:23,040 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:23,040 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-18 07:05:23,041 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 07:05:23,041 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 07:05:23,042 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-18 07:05:23,042 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:23,042 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-18 07:05:23,044 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table 2023-07-18 07:05:23,044 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table 2023-07-18 07:05:23,044 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-18 07:05:23,045 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:23,046 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740 2023-07-18 07:05:23,049 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740 2023-07-18 07:05:23,051 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-18 07:05:23,058 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-18 07:05:23,059 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-18 07:05:23,111 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/.tmp/info/d20f141ad6814706b560aca1a53ce251 2023-07-18 07:05:23,169 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/.tmp/table/808f05c5392a425ab7b0dfcab882ae35 2023-07-18 07:05:23,178 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/.tmp/info/d20f141ad6814706b560aca1a53ce251 as hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info/d20f141ad6814706b560aca1a53ce251 2023-07-18 07:05:23,186 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info/d20f141ad6814706b560aca1a53ce251, entries=20, sequenceid=13, filesize=6.9 K 2023-07-18 07:05:23,189 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/.tmp/table/808f05c5392a425ab7b0dfcab882ae35 as hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table/808f05c5392a425ab7b0dfcab882ae35 2023-07-18 07:05:23,198 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table/808f05c5392a425ab7b0dfcab882ae35, entries=4, sequenceid=13, filesize=4.7 K 2023-07-18 07:05:23,198 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 139ms, sequenceid=13, compaction requested=false; wal=null 2023-07-18 07:05:23,200 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-18 07:05:23,202 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-18 07:05:23,206 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-18 07:05:23,208 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-18 07:05:23,210 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-18 07:05:23,211 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70550053, jitterRate=0.05127771198749542}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-18 07:05:23,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-18 07:05:23,213 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689663923004 2023-07-18 07:05:23,215 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-18 07:05:23,215 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-18 07:05:23,216 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43389,1689663918718, state=OPEN 2023-07-18 07:05:23,217 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 07:05:23,217 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 07:05:23,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-18 07:05:23,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43389,1689663918718 in 367 msec 2023-07-18 07:05:23,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-18 07:05:23,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 524 msec 2023-07-18 07:05:23,223 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,42631,1689663918670 had 2 regions 2023-07-18 07:05:23,224 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42631,1689663918670, splitWal=true, meta=true, isMeta: false 2023-07-18 07:05:23,226 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,42631,1689663918670 WAL count=1, meta=false 2023-07-18 07:05:23,226 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068}] 2023-07-18 07:05:23,228 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:23,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068, worker=jenkins-hbase4.apache.org,43389,1689663918718}] 2023-07-18 07:05:23,382 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43389] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-18 07:05:23,396 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068, size=1.1 K (1148bytes) 2023-07-18 07:05:23,396 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068 2023-07-18 07:05:23,397 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068 after 0ms 2023-07-18 07:05:23,400 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068 took 4ms 2023-07-18 07:05:23,403 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for f8db4d5f36fb837f8dd2c3c681e31588: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-18 07:05:23,405 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068 so closing down 2023-07-18 07:05:23,405 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-18 07:05:23,413 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068.temp 2023-07-18 07:05:23,414 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-18 07:05:23,431 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-18 07:05:23,433 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068.temp to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000013 2023-07-18 07:05:23,433 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 33 ms; skipped=1; WAL=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-18 07:05:23,434 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068, journal: Splitting hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068, size=1.1 K (1148bytes) at 1689663923396Finishing writing output for hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068 so closing down at 1689663923405 (+9 ms)Creating recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068.temp at 1689663923414 (+9 ms)3 split writer threads finished at 1689663923414Closed recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689663923431 (+17 ms)Rename recovered edits hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068.temp to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000013 at 1689663923433 (+2 ms)Processed 11 edits across 1 Regions in 33 ms; skipped=1; WAL=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689663923434 (+1 ms) 2023-07-18 07:05:23,434 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068 2023-07-18 07:05:23,434 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-18 07:05:23,434 DEBUG [hconnection-0xc8a45ae-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42631 this server is in the failed servers list 2023-07-18 07:05:23,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.HMaster(3641): Remote procedure done, pid=18 2023-07-18 07:05:23,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] wal.WALSplitUtil(141): Moved hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting/jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068 to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs 2023-07-18 07:05:23,441 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-18 07:05:23,441 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068, worker=jenkins-hbase4.apache.org,43389,1689663918718 in 209 msec 2023-07-18 07:05:23,442 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:23,445 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-18 07:05:23,445 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C42631%2C1689663918670.1689663920068, worker=jenkins-hbase4.apache.org,43389,1689663918718 in 216 msec 2023-07-18 07:05:23,447 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,42631,1689663918670-splitting dir is empty, no logs to split. 2023-07-18 07:05:23,447 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,42631,1689663918670 WAL count=0, meta=false 2023-07-18 07:05:23,447 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42631,1689663918670 WAL splitting is done? wals=0, meta=false 2023-07-18 07:05:23,449 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN}] 2023-07-18 07:05:23,450 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN 2023-07-18 07:05:23,451 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 07:05:23,602 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=f8db4d5f36fb837f8dd2c3c681e31588, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:23,602 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689663923602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689663923602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689663923602"}]},"ts":"1689663923602"} 2023-07-18 07:05:23,603 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42631 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42631 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 07:05:23,604 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42631 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42631 2023-07-18 07:05:23,641 DEBUG [hconnection-0xc8a45ae-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42631 this server is in the failed servers list 2023-07-18 07:05:23,709 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure f8db4d5f36fb837f8dd2c3c681e31588, server=jenkins-hbase4.apache.org,43389,1689663918718}] 2023-07-18 07:05:23,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:23,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f8db4d5f36fb837f8dd2c3c681e31588, NAME => 't,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:23,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:23,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:23,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:23,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:23,870 INFO [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:23,871 DEBUG [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/f 2023-07-18 07:05:23,871 DEBUG [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/f 2023-07-18 07:05:23,872 INFO [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f8db4d5f36fb837f8dd2c3c681e31588 columnFamilyName f 2023-07-18 07:05:23,873 INFO [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] regionserver.HStore(310): Store=f8db4d5f36fb837f8dd2c3c681e31588/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:23,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:23,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:23,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000013 2023-07-18 07:05:23,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000013 2023-07-18 07:05:23,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f8db4d5f36fb837f8dd2c3c681e31588 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-18 07:05:23,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/.tmp/f/55445a2e45db4b06aa978b3f05210b70 2023-07-18 07:05:23,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/.tmp/f/55445a2e45db4b06aa978b3f05210b70 as hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/f/55445a2e45db4b06aa978b3f05210b70 2023-07-18 07:05:23,916 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/f/55445a2e45db4b06aa978b3f05210b70, entries=10, sequenceid=13, filesize=4.8 K 2023-07-18 07:05:23,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for f8db4d5f36fb837f8dd2c3c681e31588 in 35ms, sequenceid=13, compaction requested=false; wal=null 2023-07-18 07:05:23,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-18 07:05:23,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/0000000000000000013 2023-07-18 07:05:23,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:23,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-18 07:05:23,928 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f8db4d5f36fb837f8dd2c3c681e31588; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71818606, jitterRate=0.07018062472343445}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 07:05:23,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f8db4d5f36fb837f8dd2c3c681e31588: 2023-07-18 07:05:23,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588., pid=20, masterSystemTime=1689663923861 2023-07-18 07:05:23,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:23,931 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:23,932 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=f8db4d5f36fb837f8dd2c3c681e31588, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:23,932 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689663923932"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689663923932"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689663923932"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689663923932"}]},"ts":"1689663923932"} 2023-07-18 07:05:23,937 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-18 07:05:23,937 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure f8db4d5f36fb837f8dd2c3c681e31588, server=jenkins-hbase4.apache.org,43389,1689663918718 in 226 msec 2023-07-18 07:05:23,939 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-07-18 07:05:23,939 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,42631,1689663918670 after splitting done 2023-07-18 07:05:23,939 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN in 488 msec 2023-07-18 07:05:23,939 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,42631,1689663918670 from processing; numProcessing=0 2023-07-18 07:05:23,942 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,42631,1689663918670, splitWal=true, meta=true in 1.5410 sec 2023-07-18 07:05:23,955 DEBUG [Listener at localhost/37223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x243942ca to 127.0.0.1:50929 2023-07-18 07:05:23,955 DEBUG [Listener at localhost/37223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:23,955 INFO [Listener at localhost/37223] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-18 07:05:23,968 INFO [Listener at localhost/37223] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=271 (was 294), OpenFileDescriptor=569 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=130 (was 130), ProcessCount=169 (was 169), AvailableMemoryMB=6141 (was 6149) 2023-07-18 07:05:23,979 INFO [Listener at localhost/37223] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=271, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=130, ProcessCount=169, AvailableMemoryMB=6140 2023-07-18 07:05:23,980 INFO [Listener at localhost/37223] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:42631 2023-07-18 07:05:23,992 INFO [Listener at localhost/37223] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-18 07:05:23,992 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:23,993 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:23,993 INFO [Listener at localhost/37223] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-18 07:05:23,993 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-18 07:05:23,993 INFO [Listener at localhost/37223] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-18 07:05:23,993 INFO [Listener at localhost/37223] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-18 07:05:23,994 INFO [Listener at localhost/37223] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35743 2023-07-18 07:05:23,995 INFO [Listener at localhost/37223] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-18 07:05:23,996 DEBUG [Listener at localhost/37223] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-18 07:05:23,997 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:23,999 INFO [Listener at localhost/37223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-18 07:05:24,000 INFO [Listener at localhost/37223] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35743 connecting to ZooKeeper ensemble=127.0.0.1:50929 2023-07-18 07:05:24,004 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:357430x0, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-18 07:05:24,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:35743-0x101773fcd8e0008 connected 2023-07-18 07:05:24,006 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(162): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-18 07:05:24,007 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(162): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-18 07:05:24,008 DEBUG [Listener at localhost/37223] zookeeper.ZKUtil(164): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-18 07:05:24,008 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35743 2023-07-18 07:05:24,009 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35743 2023-07-18 07:05:24,009 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35743 2023-07-18 07:05:24,009 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35743 2023-07-18 07:05:24,010 DEBUG [Listener at localhost/37223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35743 2023-07-18 07:05:24,011 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(951): ClusterId : 7ed3d068-ed92-4590-a281-cc5ca5fc66fb 2023-07-18 07:05:24,012 DEBUG [RS:2;jenkins-hbase4:35743] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-18 07:05:24,014 DEBUG [RS:2;jenkins-hbase4:35743] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-18 07:05:24,014 DEBUG [RS:2;jenkins-hbase4:35743] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-18 07:05:24,016 DEBUG [RS:2;jenkins-hbase4:35743] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-18 07:05:24,017 DEBUG [RS:2;jenkins-hbase4:35743] zookeeper.ReadOnlyZKClient(139): Connect 0x0f9da9c7 to 127.0.0.1:50929 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 07:05:24,020 DEBUG [RS:2;jenkins-hbase4:35743] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15a4644, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 07:05:24,021 DEBUG [RS:2;jenkins-hbase4:35743] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@565c2b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-18 07:05:24,029 DEBUG [RS:2;jenkins-hbase4:35743] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:35743 2023-07-18 07:05:24,029 INFO [RS:2;jenkins-hbase4:35743] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-18 07:05:24,029 INFO [RS:2;jenkins-hbase4:35743] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-18 07:05:24,029 DEBUG [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1022): About to register with Master. 2023-07-18 07:05:24,030 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,46813,1689663917527 with isa=jenkins-hbase4.apache.org/172.31.14.131:35743, startcode=1689663923992 2023-07-18 07:05:24,030 DEBUG [RS:2;jenkins-hbase4:35743] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-18 07:05:24,033 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39845, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-18 07:05:24,033 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46813] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:24,034 DEBUG [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9 2023-07-18 07:05:24,034 DEBUG [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:37543 2023-07-18 07:05:24,034 DEBUG [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-18 07:05:24,037 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:24,037 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:24,037 DEBUG [RS:2;jenkins-hbase4:35743] zookeeper.ZKUtil(162): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:24,038 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35743,1689663923992] 2023-07-18 07:05:24,038 WARN [RS:2;jenkins-hbase4:35743] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-18 07:05:24,038 INFO [RS:2;jenkins-hbase4:35743] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 07:05:24,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:24,038 DEBUG [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1948): logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:24,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:24,046 DEBUG [RS:2;jenkins-hbase4:35743] zookeeper.ZKUtil(162): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:24,046 DEBUG [RS:2;jenkins-hbase4:35743] zookeeper.ZKUtil(162): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:24,048 DEBUG [RS:2;jenkins-hbase4:35743] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-18 07:05:24,048 INFO [RS:2;jenkins-hbase4:35743] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-18 07:05:24,050 INFO [RS:2;jenkins-hbase4:35743] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-18 07:05:24,051 INFO [RS:2;jenkins-hbase4:35743] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-18 07:05:24,051 INFO [RS:2;jenkins-hbase4:35743] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:24,051 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-18 07:05:24,052 INFO [RS:2;jenkins-hbase4:35743] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:24,053 DEBUG [RS:2;jenkins-hbase4:35743] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-18 07:05:24,054 INFO [RS:2;jenkins-hbase4:35743] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:24,054 INFO [RS:2;jenkins-hbase4:35743] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:24,055 INFO [RS:2;jenkins-hbase4:35743] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:24,073 INFO [RS:2;jenkins-hbase4:35743] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-18 07:05:24,073 INFO [RS:2;jenkins-hbase4:35743] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35743,1689663923992-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-18 07:05:24,086 INFO [RS:2;jenkins-hbase4:35743] regionserver.Replication(203): jenkins-hbase4.apache.org,35743,1689663923992 started 2023-07-18 07:05:24,086 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,35743,1689663923992, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35743, sessionid=0x101773fcd8e0008 2023-07-18 07:05:24,086 INFO [Listener at localhost/37223] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:35743,5,FailOnTimeoutGroup] 2023-07-18 07:05:24,086 DEBUG [RS:2;jenkins-hbase4:35743] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-18 07:05:24,086 DEBUG [RS:2;jenkins-hbase4:35743] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:24,086 DEBUG [RS:2;jenkins-hbase4:35743] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35743,1689663923992' 2023-07-18 07:05:24,086 DEBUG [RS:2;jenkins-hbase4:35743] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-18 07:05:24,086 INFO [Listener at localhost/37223] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:24,086 INFO [Listener at localhost/37223] client.TestScannerTimeout(177): START ************ test3686b 2023-07-18 07:05:24,087 DEBUG [RS:2;jenkins-hbase4:35743] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-18 07:05:24,087 DEBUG [RS:2;jenkins-hbase4:35743] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-18 07:05:24,087 DEBUG [RS:2;jenkins-hbase4:35743] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-18 07:05:24,087 DEBUG [RS:2;jenkins-hbase4:35743] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:24,087 DEBUG [RS:2;jenkins-hbase4:35743] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35743,1689663923992' 2023-07-18 07:05:24,087 DEBUG [RS:2;jenkins-hbase4:35743] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-18 07:05:24,087 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42631 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42631 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 07:05:24,088 DEBUG [RS:2;jenkins-hbase4:35743] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-18 07:05:24,088 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42631 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42631 2023-07-18 07:05:24,090 DEBUG [RS:2;jenkins-hbase4:35743] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-18 07:05:24,090 INFO [RS:2;jenkins-hbase4:35743] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-18 07:05:24,090 INFO [RS:2;jenkins-hbase4:35743] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-18 07:05:24,191 DEBUG [hconnection-0x5ab85974-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:24,193 INFO [RS:2;jenkins-hbase4:35743] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35743%2C1689663923992, suffix=, logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,35743,1689663923992, archiveDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs, maxLogs=32 2023-07-18 07:05:24,194 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:24,197 DEBUG [Listener at localhost/37223] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-18 07:05:24,198 DEBUG [Listener at localhost/37223] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:24,198 DEBUG [Listener at localhost/37223] zookeeper.ReadOnlyZKClient(139): Connect 0x62f01f60 to 127.0.0.1:50929 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 07:05:24,204 DEBUG [Listener at localhost/37223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f812485, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 07:05:24,210 DEBUG [hconnection-0x4d07ef26-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:24,214 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41510, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:24,220 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK] 2023-07-18 07:05:24,220 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK] 2023-07-18 07:05:24,223 INFO [RS:2;jenkins-hbase4:35743] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,35743,1689663923992/jenkins-hbase4.apache.org%2C35743%2C1689663923992.1689663924195 2023-07-18 07:05:24,224 DEBUG [RS:2;jenkins-hbase4:35743] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK], DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK]] 2023-07-18 07:05:25,837 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-18 07:05:25,898 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-18 07:05:27,683 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-18 07:05:27,683 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-18 07:05:39,227 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 7656054787401580552 expired clientIPAndPort=172.31.14.131:41510, userName=jenkins, regionInfo=t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:41,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43389] regionserver.RSRpcServices(3094): Client tried to access missing scanner 7656054787401580552 2023-07-18 07:05:41,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43389] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:41510 deadline: 1689664001229, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '7656054787401580552'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-18 07:05:41,231 DEBUG [Listener at localhost/37223] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '7656054787401580552'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '7656054787401580552'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '7656054787401580552'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-18 07:05:41,235 DEBUG [Listener at localhost/37223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62f01f60 to 127.0.0.1:50929 2023-07-18 07:05:41,235 DEBUG [Listener at localhost/37223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:41,235 INFO [Listener at localhost/37223] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-18 07:05:41,251 INFO [Listener at localhost/37223] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=296 (was 271) Potentially hanging thread: RS:2;jenkins-hbase4:35743-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9-prefix:jenkins-hbase4.apache.org,35743,1689663923992 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50929@0x62f01f60-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:2;jenkins-hbase4:35743 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1725844787-172.31.14.131-1689663914536:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50929@0x62f01f60-SendThread(127.0.0.1:50929) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50929@0x0f9da9c7-SendThread(127.0.0.1:50929) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1725844787-172.31.14.131-1689663914536:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50929@0x0f9da9c7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/527824175.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase4:35743Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1816358650_17 at /127.0.0.1:37876 [Receiving block BP-1725844787-172.31.14.131-1689663914536:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50929@0x62f01f60 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/527824175.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:37543 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1816358650_17 at /127.0.0.1:56230 [Receiving block BP-1725844787-172.31.14.131-1689663914536:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50929@0x0f9da9c7-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=35743 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=573 (was 569) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=117 (was 130), ProcessCount=169 (was 169), AvailableMemoryMB=6479 (was 6140) - AvailableMemoryMB LEAK? - 2023-07-18 07:05:41,265 INFO [Listener at localhost/37223] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=296, OpenFileDescriptor=573, MaxFileDescriptor=60000, SystemLoadAverage=117, ProcessCount=169, AvailableMemoryMB=6478 2023-07-18 07:05:41,265 INFO [Listener at localhost/37223] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:42631 2023-07-18 07:05:41,265 INFO [Listener at localhost/37223] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,42631,1689663918670 2023-07-18 07:05:41,265 INFO [Listener at localhost/37223] client.TestScannerTimeout(104): START************ test2772 2023-07-18 07:05:41,269 DEBUG [Listener at localhost/37223] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-18 07:05:41,269 DEBUG [Listener at localhost/37223] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:41,270 DEBUG [Listener at localhost/37223] zookeeper.ReadOnlyZKClient(139): Connect 0x04a38435 to 127.0.0.1:50929 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-18 07:05:41,275 DEBUG [Listener at localhost/37223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bee89e8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-18 07:05:41,275 ERROR [Listener at localhost/37223] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,43389,1689663918718: die! ***** 2023-07-18 07:05:41,275 ERROR [Listener at localhost/37223] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-18 07:05:41,275 DEBUG [Listener at localhost/37223] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-18 07:05:41,276 DEBUG [Listener at localhost/37223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-18 07:05:41,283 DEBUG [Listener at localhost/37223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-18 07:05:41,285 DEBUG [Listener at localhost/37223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-18 07:05:41,300 INFO [Listener at localhost/37223] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 963641344, "init": 513802240, "max": 2051014656, "used": 298539240 }, "NonHeapMemoryUsage": { "committed": 124280832, "init": 2555904, "max": -1, "used": 121714552 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 12588, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 5, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16910 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:50929", "tag.serverName": "jenkins-hbase4.apache.org,43389,1689663918718", "tag.clusterId": "7ed3d068-ed92-4590-a281-cc5ca5fc66fb", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 3, "maxStoreFileCount": 1, "memStoreSize": 912, "storeFileSize": 16785, "maxStoreFileAge": 16771, "minStoreFileAge": 15983, "avgStoreFileAge": 16489, "numReferenceFiles": 0, "regionServerStartTime": 1689663918718, "averageRegionSize": 5899, "storeFileIndexSize": 1048, "staticIndexSize": 184, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1787, "localBytesRead": 1787, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819799248, "blockCacheCount": 3, "blockCacheDataBlockCount": 3, "blockCacheSize": 606640, "blockCacheCountHitPercent": 85.0, "blockCacheExpressHitPercent": 85.0, "l1CacheSize": 606640, "l1CacheFreeSize": 819799248, "l1CacheCount": 3, "l1CacheEvictionCount": 0, "l1CacheHitCount": 17, "l1CacheMissCount": 3, "l1CacheHitRatio": 0.85, "l1CacheMissRatio": 0.15, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 7076, "ByteBuffAllocatorPoolAllocationBytes": 1131520, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 23, "totalRowActionRequestCount": 28, "readRequestCount": 24, "filteredReadRequestCount": 0, "writeRequestCount": 4, "rpcGetRequestCount": 8, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 11, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 4, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 17, "blockCacheHitCountPrimary": 17, "blockCacheHitCachingCount": 17, "blockCacheMissCount": 3, "blockCacheMissCountPrimary": 3, "blockCacheMissCachingCount": 3, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 3, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 17, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 34, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 2664, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 10, "ScanTime_min": 1, "ScanTime_max": 1, "ScanTime_mean": 1, "ScanTime_25th_percentile": 1, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 4, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 7, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 16785, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 10, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 155, "ScanSize_median": 156, "ScanSize_75th_percentile": 653, "ScanSize_90th_percentile": 653, "ScanSize_95th_percentile": 653, "ScanSize_98th_percentile": 653, "ScanSize_99th_percentile": 653, "ScanSize_99.9th_percentile": 653, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 2664, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 52, "ServerReadQueryPerSecond_mean_rate": 2.4297999336756484, "ServerReadQueryPerSecond_1min_rate": 7.164967204256925, "ServerReadQueryPerSecond_5min_rate": 8.751310705406569, "ServerReadQueryPerSecond_15min_rate": 9.047937375158883, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.1214460777502189, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-18 07:05:41,302 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46813] master.MasterRpcServices(612): jenkins-hbase4.apache.org,43389,1689663918718 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,43389,1689663918718: die! ***** 2023-07-18 07:05:41,304 INFO [Listener at localhost/37223] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,43389,1689663918718' ***** 2023-07-18 07:05:41,305 INFO [Listener at localhost/37223] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-18 07:05:41,305 INFO [RS:1;jenkins-hbase4:43389] regionserver.HeapMemoryManager(220): Stopping 2023-07-18 07:05:41,305 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-18 07:05:41,305 INFO [RS:1;jenkins-hbase4:43389] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-18 07:05:41,305 INFO [RS:1;jenkins-hbase4:43389] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-18 07:05:41,305 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(3305): Received CLOSE for a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:41,306 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(3305): Received CLOSE for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:41,306 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:41,306 DEBUG [RS:1;jenkins-hbase4:43389] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1bbbc2ce to 127.0.0.1:50929 2023-07-18 07:05:41,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a53d19b08d23f40ffad9dc6f21a56d09, disabling compactions & flushes 2023-07-18 07:05:41,306 DEBUG [RS:1;jenkins-hbase4:43389] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:41,306 INFO [RS:1;jenkins-hbase4:43389] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-18 07:05:41,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:41,307 INFO [RS:1;jenkins-hbase4:43389] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-18 07:05:41,307 INFO [RS:1;jenkins-hbase4:43389] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-18 07:05:41,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:41,307 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-18 07:05:41,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. after waiting 0 ms 2023-07-18 07:05:41,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:41,308 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-18 07:05:41,308 DEBUG [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1478): Online Regions={a53d19b08d23f40ffad9dc6f21a56d09=hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09., 1588230740=hbase:meta,,1.1588230740, f8db4d5f36fb837f8dd2c3c681e31588=t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.} 2023-07-18 07:05:41,308 DEBUG [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1504): Waiting on 1588230740, a53d19b08d23f40ffad9dc6f21a56d09, f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:41,312 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-18 07:05:41,312 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-18 07:05:41,312 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:41,312 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-18 07:05:41,313 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-18 07:05:41,313 DEBUG [hconnection-0x7fffe7ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:41,313 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-18 07:05:41,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:41,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a53d19b08d23f40ffad9dc6f21a56d09: 2023-07-18 07:05:41,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:41,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f8db4d5f36fb837f8dd2c3c681e31588, disabling compactions & flushes 2023-07-18 07:05:41,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:41,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:41,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. after waiting 0 ms 2023-07-18 07:05:41,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:41,319 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:41,319 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43389] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43389,1689663918718 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 07:05:41,320 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43389] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:39262 deadline: 1689664001319, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43389,1689663918718 aborting 2023-07-18 07:05:41,331 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:41,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f8db4d5f36fb837f8dd2c3c681e31588: 2023-07-18 07:05:41,332 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 834 in region hbase:meta,,1.1588230740 2023-07-18 07:05:41,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:41,332 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-18 07:05:41,333 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-18 07:05:41,333 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-18 07:05:41,333 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-18 07:05:41,398 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-18 07:05:41,433 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43389] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43389,1689663918718 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-18 07:05:41,433 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43389] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:39262 deadline: 1689664001433, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43389,1689663918718 aborting 2023-07-18 07:05:41,508 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,43389,1689663918718; all regions closed. 2023-07-18 07:05:41,528 DEBUG [RS:1;jenkins-hbase4:43389] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:41,528 INFO [RS:1;jenkins-hbase4:43389] regionserver.LeaseManager(133): Closed leases 2023-07-18 07:05:41,528 INFO [RS:1;jenkins-hbase4:43389] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-18 07:05:41,529 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-18 07:05:41,529 INFO [RS:1;jenkins-hbase4:43389] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43389 2023-07-18 07:05:41,533 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:41,533 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:41,533 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 2023-07-18 07:05:41,533 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:41,533 ERROR [Listener at localhost/37223-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f6a55b3 rejected from java.util.concurrent.ThreadPoolExecutor@3b55c686[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1374) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:603) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-07-18 07:05:41,533 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:41,534 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43389,1689663918718] 2023-07-18 07:05:41,534 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43389,1689663918718; numProcessing=1 2023-07-18 07:05:41,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:41,535 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,43389,1689663918718 znode expired, triggering replicatorRemoved event 2023-07-18 07:05:41,536 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43389,1689663918718 already deleted, retry=false 2023-07-18 07:05:41,536 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,43389,1689663918718 on jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:41,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:41,538 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=21, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,43389,1689663918718, splitWal=true, meta=true 2023-07-18 07:05:41,538 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=21 for jenkins-hbase4.apache.org,43389,1689663918718 (carryingMeta=true) jenkins-hbase4.apache.org,43389,1689663918718/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@6be54bc1[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-18 07:05:41,540 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=21, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43389,1689663918718, splitWal=true, meta=true 2023-07-18 07:05:41,541 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43389,1689663918718, splitWal=true, meta=true, isMeta: true 2023-07-18 07:05:41,550 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting 2023-07-18 07:05:41,552 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,43389,1689663918718 WAL count=1, meta=true 2023-07-18 07:05:41,552 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta}] 2023-07-18 07:05:41,555 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:41,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta, worker=jenkins-hbase4.apache.org,35743,1689663923992}] 2023-07-18 07:05:41,635 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 07:05:41,635 INFO [RS:1;jenkins-hbase4:43389] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,43389,1689663918718; zookeeper connection closed. 2023-07-18 07:05:41,635 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:43389-0x101773fcd8e0002, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 07:05:41,635 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5e2069e7] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5e2069e7 2023-07-18 07:05:41,637 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43389 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43389 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 07:05:41,638 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43389 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43389 2023-07-18 07:05:41,709 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:41,709 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-18 07:05:41,711 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49956, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-18 07:05:41,711 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35743] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-18 07:05:41,727 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta, size=1.3 K (1341bytes) 2023-07-18 07:05:41,727 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta 2023-07-18 07:05:41,728 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta after 0ms 2023-07-18 07:05:41,734 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta took 7ms 2023-07-18 07:05:41,740 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-18 07:05:41,742 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta so closing down 2023-07-18 07:05:41,742 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-18 07:05:41,747 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta.temp 2023-07-18 07:05:41,747 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-18 07:05:41,762 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-18 07:05:41,764 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta.temp to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-18 07:05:41,764 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 3 edits across 1 Regions in 29 ms; skipped=1; WAL=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false 2023-07-18 07:05:41,764 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta, journal: Splitting hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta, size=1.3 K (1341bytes) at 1689663941727Finishing writing output for hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta so closing down at 1689663941742 (+15 ms)Creating recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta.temp at 1689663941747 (+5 ms)3 split writer threads finished at 1689663941747Closed recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689663941762 (+15 ms)Rename recovered edits hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta.temp to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 at 1689663941764 (+2 ms)Processed 3 edits across 1 Regions in 29 ms; skipped=1; WAL=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false at 1689663941764 2023-07-18 07:05:41,765 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta 2023-07-18 07:05:41,765 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-18 07:05:41,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.HMaster(3641): Remote procedure done, pid=23 2023-07-18 07:05:41,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] wal.WALSplitUtil(141): Moved hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs 2023-07-18 07:05:41,773 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-18 07:05:41,773 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta, worker=jenkins-hbase4.apache.org,35743,1689663923992 in 214 msec 2023-07-18 07:05:41,774 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:41,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-07-18 07:05:41,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43389%2C1689663918718.meta.1689663923012.meta, worker=jenkins-hbase4.apache.org,35743,1689663923992 in 222 msec 2023-07-18 07:05:41,779 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting dir is empty, no logs to split. 2023-07-18 07:05:41,780 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,43389,1689663918718 WAL count=0, meta=true 2023-07-18 07:05:41,780 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43389,1689663918718 WAL splitting is done? wals=0, meta=true 2023-07-18 07:05:41,781 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-18 07:05:41,782 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-18 07:05:41,783 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 07:05:41,933 DEBUG [jenkins-hbase4:46813] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-18 07:05:41,934 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35743,1689663923992, state=OPENING 2023-07-18 07:05:41,939 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 07:05:41,939 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 07:05:41,939 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35743,1689663923992}] 2023-07-18 07:05:41,940 WARN [ReadOnlyZKClient-127.0.0.1:50929@0x04a38435] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-18 07:05:41,941 DEBUG [hconnection-0x7fffe7ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:41,944 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:41,945 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=35743] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:49964 deadline: 1689664001944, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:42,098 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-18 07:05:42,098 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-18 07:05:42,101 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35743%2C1689663923992.meta, suffix=.meta, logDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,35743,1689663923992, archiveDir=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs, maxLogs=32 2023-07-18 07:05:42,121 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK] 2023-07-18 07:05:42,122 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK] 2023-07-18 07:05:42,125 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,35743,1689663923992/jenkins-hbase4.apache.org%2C35743%2C1689663923992.meta.1689663942102.meta 2023-07-18 07:05:42,125 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46661,DS-1326d674-ade8-4b08-823f-06fe13d00ca3,DISK], DatanodeInfoWithStorage[127.0.0.1:36191,DS-f3ff6f9d-e206-4bc1-8547-23aa79aa7273,DISK]] 2023-07-18 07:05:42,126 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:42,126 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-18 07:05:42,126 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-18 07:05:42,126 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-18 07:05:42,126 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-18 07:05:42,126 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:42,126 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-18 07:05:42,126 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-18 07:05:42,128 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-18 07:05:42,130 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info 2023-07-18 07:05:42,130 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info 2023-07-18 07:05:42,131 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-18 07:05:42,142 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info/d20f141ad6814706b560aca1a53ce251 2023-07-18 07:05:42,143 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:42,143 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-18 07:05:42,145 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 07:05:42,145 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/rep_barrier 2023-07-18 07:05:42,145 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-18 07:05:42,146 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:42,146 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-18 07:05:42,148 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table 2023-07-18 07:05:42,148 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table 2023-07-18 07:05:42,149 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-18 07:05:42,161 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/table/808f05c5392a425ab7b0dfcab882ae35 2023-07-18 07:05:42,161 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:42,162 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740 2023-07-18 07:05:42,165 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740 2023-07-18 07:05:42,166 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-18 07:05:42,171 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 8, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=17, path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-18 07:05:42,171 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=834 B heapSize=2.16 KB 2023-07-18 07:05:42,186 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=834 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/.tmp/info/0ed30183152c4d8dabca054862433dfd 2023-07-18 07:05:42,194 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/.tmp/info/0ed30183152c4d8dabca054862433dfd as hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info/0ed30183152c4d8dabca054862433dfd 2023-07-18 07:05:42,201 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info/0ed30183152c4d8dabca054862433dfd, entries=8, sequenceid=17, filesize=5.5 K 2023-07-18 07:05:42,202 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~834 B/834, heapSize ~1.64 KB/1680, currentSize=0 B/0 for 1588230740 in 31ms, sequenceid=17, compaction requested=false; wal=null 2023-07-18 07:05:42,203 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-18 07:05:42,206 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-18 07:05:42,207 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-18 07:05:42,211 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=13 2023-07-18 07:05:42,212 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=18; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64604447, jitterRate=-0.037318721413612366}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-18 07:05:42,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-18 07:05:42,213 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=25, masterSystemTime=1689663942092 2023-07-18 07:05:42,215 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-18 07:05:42,216 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-18 07:05:42,216 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35743,1689663923992, state=OPEN 2023-07-18 07:05:42,218 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-18 07:05:42,218 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-18 07:05:42,221 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-18 07:05:42,221 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35743,1689663923992 in 279 msec 2023-07-18 07:05:42,223 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-18 07:05:42,223 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,43389,1689663918718 had 3 regions 2023-07-18 07:05:42,223 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 440 msec 2023-07-18 07:05:42,224 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43389,1689663918718, splitWal=true, meta=true, isMeta: false 2023-07-18 07:05:42,227 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,43389,1689663918718 WAL count=1, meta=false 2023-07-18 07:05:42,227 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068}] 2023-07-18 07:05:42,229 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:42,230 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068, worker=jenkins-hbase4.apache.org,35743,1689663923992}] 2023-07-18 07:05:42,383 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=35743] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=27 2023-07-18 07:05:42,403 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068, size=899 (899bytes) 2023-07-18 07:05:42,403 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068 2023-07-18 07:05:42,405 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068 after 2ms 2023-07-18 07:05:42,412 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068 took 9ms 2023-07-18 07:05:42,425 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for a53d19b08d23f40ffad9dc6f21a56d09: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-18 07:05:42,428 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for f8db4d5f36fb837f8dd2c3c681e31588: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-18 07:05:42,428 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068 so closing down 2023-07-18 07:05:42,428 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-18 07:05:42,431 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068.temp 2023-07-18 07:05:42,431 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-18 07:05:42,459 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-18 07:05:42,462 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068.temp to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000005 2023-07-18 07:05:42,463 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 44 ms; skipped=2; WAL=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068, size=899, length=899, corrupted=false, cancelled=false 2023-07-18 07:05:42,463 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068, journal: Splitting hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068, size=899 (899bytes) at 1689663942403Finishing writing output for hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068 so closing down at 1689663942428 (+25 ms)Creating recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068.temp at 1689663942431 (+3 ms)3 split writer threads finished at 1689663942431Closed recovered edits writer path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689663942459 (+28 ms)Rename recovered edits hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068.temp to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000005 at 1689663942462 (+3 ms)Processed 4 edits across 1 Regions in 44 ms; skipped=2; WAL=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068, size=899, length=899, corrupted=false, cancelled=false at 1689663942463 (+1 ms) 2023-07-18 07:05:42,463 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068 2023-07-18 07:05:42,463 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=27 2023-07-18 07:05:42,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] master.HMaster(3641): Remote procedure done, pid=27 2023-07-18 07:05:42,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46813] wal.WALSplitUtil(141): Moved hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting/jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068 to hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs 2023-07-18 07:05:42,467 DEBUG [hconnection-0x7fffe7ab-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43389 this server is in the failed servers list 2023-07-18 07:05:42,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-18 07:05:42,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068, worker=jenkins-hbase4.apache.org,35743,1689663923992 in 239 msec 2023-07-18 07:05:42,473 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:42,476 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=21 2023-07-18 07:05:42,476 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43389%2C1689663918718.1689663920068, worker=jenkins-hbase4.apache.org,35743,1689663923992 in 246 msec 2023-07-18 07:05:42,478 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/WALs/jenkins-hbase4.apache.org,43389,1689663918718-splitting dir is empty, no logs to split. 2023-07-18 07:05:42,478 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,43389,1689663918718 WAL count=0, meta=false 2023-07-18 07:05:42,478 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43389,1689663918718 WAL splitting is done? wals=0, meta=false 2023-07-18 07:05:42,480 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN}, {pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a53d19b08d23f40ffad9dc6f21a56d09, ASSIGN}] 2023-07-18 07:05:42,481 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a53d19b08d23f40ffad9dc6f21a56d09, ASSIGN 2023-07-18 07:05:42,482 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN 2023-07-18 07:05:42,483 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a53d19b08d23f40ffad9dc6f21a56d09, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 07:05:42,483 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-18 07:05:42,576 DEBUG [hconnection-0x7fffe7ab-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43389 this server is in the failed servers list 2023-07-18 07:05:42,633 DEBUG [jenkins-hbase4:46813] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-18 07:05:42,633 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a53d19b08d23f40ffad9dc6f21a56d09, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:42,633 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=f8db4d5f36fb837f8dd2c3c681e31588, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:42,634 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689663942633"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689663942633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689663942633"}]},"ts":"1689663942633"} 2023-07-18 07:05:42,634 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689663942633"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689663942633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689663942633"}]},"ts":"1689663942633"} 2023-07-18 07:05:42,635 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43389 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43389 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-18 07:05:42,636 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43389 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43389 2023-07-18 07:05:42,738 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-18 07:05:42,739 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49966, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-18 07:05:42,742 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure a53d19b08d23f40ffad9dc6f21a56d09, server=jenkins-hbase4.apache.org,35743,1689663923992}] 2023-07-18 07:05:42,744 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE; OpenRegionProcedure f8db4d5f36fb837f8dd2c3c681e31588, server=jenkins-hbase4.apache.org,35743,1689663923992}] 2023-07-18 07:05:42,782 DEBUG [hconnection-0x7fffe7ab-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43389 this server is in the failed servers list 2023-07-18 07:05:42,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:42,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f8db4d5f36fb837f8dd2c3c681e31588, NAME => 't,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:42,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:42,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:42,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:42,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:42,909 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:42,909 INFO [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:42,909 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a53d19b08d23f40ffad9dc6f21a56d09, NAME => 'hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.', STARTKEY => '', ENDKEY => ''} 2023-07-18 07:05:42,910 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:42,910 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-18 07:05:42,910 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:42,910 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:42,911 DEBUG [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/f 2023-07-18 07:05:42,911 DEBUG [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/f 2023-07-18 07:05:42,911 INFO [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f8db4d5f36fb837f8dd2c3c681e31588 columnFamilyName f 2023-07-18 07:05:42,912 INFO [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:42,914 DEBUG [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/info 2023-07-18 07:05:42,914 DEBUG [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/info 2023-07-18 07:05:42,914 INFO [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a53d19b08d23f40ffad9dc6f21a56d09 columnFamilyName info 2023-07-18 07:05:42,915 INFO [StoreOpener-a53d19b08d23f40ffad9dc6f21a56d09-1] regionserver.HStore(310): Store=a53d19b08d23f40ffad9dc6f21a56d09/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:42,920 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:42,922 DEBUG [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] regionserver.HStore(539): loaded hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/f/55445a2e45db4b06aa978b3f05210b70 2023-07-18 07:05:42,925 INFO [StoreOpener-f8db4d5f36fb837f8dd2c3c681e31588-1] regionserver.HStore(310): Store=f8db4d5f36fb837f8dd2c3c681e31588/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-18 07:05:42,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:42,927 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:42,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:42,928 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000005 2023-07-18 07:05:42,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:42,934 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000005 2023-07-18 07:05:42,934 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a53d19b08d23f40ffad9dc6f21a56d09 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-18 07:05:42,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-18 07:05:42,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f8db4d5f36fb837f8dd2c3c681e31588; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61018430, jitterRate=-0.09075453877449036}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 07:05:42,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f8db4d5f36fb837f8dd2c3c681e31588: 2023-07-18 07:05:42,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588., pid=31, masterSystemTime=1689663942896 2023-07-18 07:05:42,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:42,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:42,944 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=f8db4d5f36fb837f8dd2c3c681e31588, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:42,945 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689663942944"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689663942944"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689663942944"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689663942944"}]},"ts":"1689663942944"} 2023-07-18 07:05:42,954 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=28 2023-07-18 07:05:42,954 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; OpenRegionProcedure f8db4d5f36fb837f8dd2c3c681e31588, server=jenkins-hbase4.apache.org,35743,1689663923992 in 205 msec 2023-07-18 07:05:42,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=t, region=f8db4d5f36fb837f8dd2c3c681e31588, ASSIGN in 474 msec 2023-07-18 07:05:42,970 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/.tmp/info/fd1378d839ed4ff791ac839c8c61a400 2023-07-18 07:05:42,981 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/.tmp/info/fd1378d839ed4ff791ac839c8c61a400 as hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/info/fd1378d839ed4ff791ac839c8c61a400 2023-07-18 07:05:42,989 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/info/fd1378d839ed4ff791ac839c8c61a400, entries=2, sequenceid=5, filesize=4.8 K 2023-07-18 07:05:42,990 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for a53d19b08d23f40ffad9dc6f21a56d09 in 56ms, sequenceid=5, compaction requested=false; wal=null 2023-07-18 07:05:42,991 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/0000000000000000005 2023-07-18 07:05:42,996 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:42,999 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-18 07:05:43,001 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a53d19b08d23f40ffad9dc6f21a56d09; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75207373, jitterRate=0.12067718803882599}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-18 07:05:43,001 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a53d19b08d23f40ffad9dc6f21a56d09: 2023-07-18 07:05:43,002 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09., pid=30, masterSystemTime=1689663942896 2023-07-18 07:05:43,005 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:43,005 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:43,005 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a53d19b08d23f40ffad9dc6f21a56d09, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:43,006 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689663943005"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689663943005"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689663943005"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689663943005"}]},"ts":"1689663943005"} 2023-07-18 07:05:43,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-07-18 07:05:43,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure a53d19b08d23f40ffad9dc6f21a56d09, server=jenkins-hbase4.apache.org,35743,1689663923992 in 267 msec 2023-07-18 07:05:43,014 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=21 2023-07-18 07:05:43,014 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,43389,1689663918718 after splitting done 2023-07-18 07:05:43,014 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a53d19b08d23f40ffad9dc6f21a56d09, ASSIGN in 531 msec 2023-07-18 07:05:43,014 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,43389,1689663918718 from processing; numProcessing=0 2023-07-18 07:05:43,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,43389,1689663918718, splitWal=true, meta=true in 1.4790 sec 2023-07-18 07:05:43,097 DEBUG [Listener at localhost/37223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04a38435 to 127.0.0.1:50929 2023-07-18 07:05:43,097 DEBUG [Listener at localhost/37223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:43,097 INFO [Listener at localhost/37223] client.TestScannerTimeout(124): END ************ test2772 2023-07-18 07:05:43,114 INFO [Listener at localhost/37223] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 296), OpenFileDescriptor=588 (was 573) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=124 (was 117) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6416 (was 6478) 2023-07-18 07:05:43,115 INFO [Listener at localhost/37223] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-18 07:05:43,115 INFO [Listener at localhost/37223] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-18 07:05:43,115 DEBUG [Listener at localhost/37223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30d0fb75 to 127.0.0.1:50929 2023-07-18 07:05:43,115 DEBUG [Listener at localhost/37223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:43,115 DEBUG [Listener at localhost/37223] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-18 07:05:43,116 DEBUG [Listener at localhost/37223] util.JVMClusterUtil(257): Found active master hash=912702398, stopped=false 2023-07-18 07:05:43,116 INFO [Listener at localhost/37223] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:43,120 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-18 07:05:43,120 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-18 07:05:43,120 INFO [Listener at localhost/37223] procedure2.ProcedureExecutor(629): Stopping 2023-07-18 07:05:43,120 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:43,120 DEBUG [Listener at localhost/37223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18a07576 to 127.0.0.1:50929 2023-07-18 07:05:43,120 DEBUG [Listener at localhost/37223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:43,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 07:05:43,121 INFO [Listener at localhost/37223] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,35743,1689663923992' ***** 2023-07-18 07:05:43,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-18 07:05:43,121 INFO [Listener at localhost/37223] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-18 07:05:43,121 INFO [RS:2;jenkins-hbase4:35743] regionserver.HeapMemoryManager(220): Stopping 2023-07-18 07:05:43,121 INFO [RS:2;jenkins-hbase4:35743] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-18 07:05:43,121 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-18 07:05:43,121 INFO [RS:2;jenkins-hbase4:35743] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-18 07:05:43,122 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(3305): Received CLOSE for a53d19b08d23f40ffad9dc6f21a56d09 2023-07-18 07:05:43,122 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(3305): Received CLOSE for f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:43,122 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:43,122 DEBUG [RS:2;jenkins-hbase4:35743] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f9da9c7 to 127.0.0.1:50929 2023-07-18 07:05:43,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a53d19b08d23f40ffad9dc6f21a56d09, disabling compactions & flushes 2023-07-18 07:05:43,122 DEBUG [RS:2;jenkins-hbase4:35743] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:43,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:43,122 INFO [RS:2;jenkins-hbase4:35743] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-18 07:05:43,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:43,123 INFO [RS:2;jenkins-hbase4:35743] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-18 07:05:43,123 INFO [RS:2;jenkins-hbase4:35743] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-18 07:05:43,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. after waiting 0 ms 2023-07-18 07:05:43,123 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-18 07:05:43,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:43,123 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-18 07:05:43,123 DEBUG [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1478): Online Regions={a53d19b08d23f40ffad9dc6f21a56d09=hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09., 1588230740=hbase:meta,,1.1588230740, f8db4d5f36fb837f8dd2c3c681e31588=t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588.} 2023-07-18 07:05:43,123 DEBUG [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1504): Waiting on 1588230740, a53d19b08d23f40ffad9dc6f21a56d09, f8db4d5f36fb837f8dd2c3c681e31588 2023-07-18 07:05:43,124 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-18 07:05:43,124 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-18 07:05:43,124 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-18 07:05:43,125 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-18 07:05:43,125 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-18 07:05:43,125 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-18 07:05:43,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/namespace/a53d19b08d23f40ffad9dc6f21a56d09/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-18 07:05:43,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:43,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a53d19b08d23f40ffad9dc6f21a56d09: 2023-07-18 07:05:43,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689663920525.a53d19b08d23f40ffad9dc6f21a56d09. 2023-07-18 07:05:43,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f8db4d5f36fb837f8dd2c3c681e31588, disabling compactions & flushes 2023-07-18 07:05:43,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:43,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:43,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. after waiting 0 ms 2023-07-18 07:05:43,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:43,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/default/t/f8db4d5f36fb837f8dd2c3c681e31588/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-18 07:05:43,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:43,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f8db4d5f36fb837f8dd2c3c681e31588: 2023-07-18 07:05:43,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689663921421.f8db4d5f36fb837f8dd2c3c681e31588. 2023-07-18 07:05:43,172 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/.tmp/info/47e9e873244242b7b3464fadae4625c0 2023-07-18 07:05:43,186 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/.tmp/info/47e9e873244242b7b3464fadae4625c0 as hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info/47e9e873244242b7b3464fadae4625c0 2023-07-18 07:05:43,195 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/info/47e9e873244242b7b3464fadae4625c0, entries=16, sequenceid=24, filesize=6.4 K 2023-07-18 07:05:43,196 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 71ms, sequenceid=24, compaction requested=true 2023-07-18 07:05:43,207 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=17 2023-07-18 07:05:43,208 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-18 07:05:43,208 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-18 07:05:43,208 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-18 07:05:43,208 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-18 07:05:43,324 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,35743,1689663923992; all regions closed. 2023-07-18 07:05:43,334 DEBUG [RS:2;jenkins-hbase4:35743] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs 2023-07-18 07:05:43,334 INFO [RS:2;jenkins-hbase4:35743] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35743%2C1689663923992.meta:.meta(num 1689663942102) 2023-07-18 07:05:43,341 DEBUG [RS:2;jenkins-hbase4:35743] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/oldWALs 2023-07-18 07:05:43,341 INFO [RS:2;jenkins-hbase4:35743] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35743%2C1689663923992:(num 1689663924195) 2023-07-18 07:05:43,341 DEBUG [RS:2;jenkins-hbase4:35743] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:43,341 INFO [RS:2;jenkins-hbase4:35743] regionserver.LeaseManager(133): Closed leases 2023-07-18 07:05:43,341 INFO [RS:2;jenkins-hbase4:35743] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-18 07:05:43,342 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-18 07:05:43,342 INFO [RS:2;jenkins-hbase4:35743] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35743 2023-07-18 07:05:43,346 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35743,1689663923992 2023-07-18 07:05:43,346 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:43,346 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-18 07:05:43,347 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35743,1689663923992] 2023-07-18 07:05:43,347 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35743,1689663923992; numProcessing=1 2023-07-18 07:05:43,349 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35743,1689663923992 already deleted, retry=false 2023-07-18 07:05:43,349 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35743,1689663923992 expired; onlineServers=0 2023-07-18 07:05:43,350 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,46813,1689663917527' ***** 2023-07-18 07:05:43,350 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-18 07:05:43,350 DEBUG [M:0;jenkins-hbase4:46813] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@775bb185, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-18 07:05:43,350 INFO [M:0;jenkins-hbase4:46813] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,46813,1689663917527 2023-07-18 07:05:43,350 INFO [M:0;jenkins-hbase4:46813] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,46813,1689663917527; all regions closed. 2023-07-18 07:05:43,350 DEBUG [M:0;jenkins-hbase4:46813] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-18 07:05:43,350 DEBUG [M:0;jenkins-hbase4:46813] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-18 07:05:43,350 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-18 07:05:43,351 DEBUG [M:0;jenkins-hbase4:46813] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-18 07:05:43,351 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689663919729] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689663919729,5,FailOnTimeoutGroup] 2023-07-18 07:05:43,351 INFO [M:0;jenkins-hbase4:46813] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-18 07:05:43,351 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689663919729] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689663919729,5,FailOnTimeoutGroup] 2023-07-18 07:05:43,351 INFO [M:0;jenkins-hbase4:46813] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-18 07:05:43,351 INFO [M:0;jenkins-hbase4:46813] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-18 07:05:43,351 DEBUG [M:0;jenkins-hbase4:46813] master.HMaster(1512): Stopping service threads 2023-07-18 07:05:43,351 INFO [M:0;jenkins-hbase4:46813] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-18 07:05:43,352 ERROR [M:0;jenkins-hbase4:46813] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1706702314) connection to localhost/127.0.0.1:37543 from jenkins,5,PEWorkerGroup] 2023-07-18 07:05:43,352 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-18 07:05:43,352 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-18 07:05:43,352 INFO [M:0;jenkins-hbase4:46813] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-18 07:05:43,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-18 07:05:43,352 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-18 07:05:43,353 DEBUG [M:0;jenkins-hbase4:46813] zookeeper.ZKUtil(398): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-18 07:05:43,353 WARN [M:0;jenkins-hbase4:46813] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-18 07:05:43,353 INFO [M:0;jenkins-hbase4:46813] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-18 07:05:43,353 INFO [M:0;jenkins-hbase4:46813] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-18 07:05:43,353 DEBUG [M:0;jenkins-hbase4:46813] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-18 07:05:43,354 INFO [M:0;jenkins-hbase4:46813] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 07:05:43,354 DEBUG [M:0;jenkins-hbase4:46813] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 07:05:43,354 DEBUG [M:0;jenkins-hbase4:46813] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-18 07:05:43,354 DEBUG [M:0;jenkins-hbase4:46813] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 07:05:43,354 INFO [M:0;jenkins-hbase4:46813] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=105.75 KB heapSize=127.45 KB 2023-07-18 07:05:43,382 INFO [M:0;jenkins-hbase4:46813] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=105.75 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/34b54048b8b5469583d3725a86191cfd 2023-07-18 07:05:43,393 DEBUG [M:0;jenkins-hbase4:46813] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/34b54048b8b5469583d3725a86191cfd as hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/34b54048b8b5469583d3725a86191cfd 2023-07-18 07:05:43,401 INFO [M:0;jenkins-hbase4:46813] regionserver.HStore(1080): Added hdfs://localhost:37543/user/jenkins/test-data/63f9c2b0-d7f2-b066-2fce-4b6bfb3350c9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/34b54048b8b5469583d3725a86191cfd, entries=31, sequenceid=244, filesize=8.4 K 2023-07-18 07:05:43,405 INFO [M:0;jenkins-hbase4:46813] regionserver.HRegion(2948): Finished flush of dataSize ~105.75 KB/108290, heapSize ~127.44 KB/130496, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 51ms, sequenceid=244, compaction requested=false 2023-07-18 07:05:43,411 INFO [M:0;jenkins-hbase4:46813] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-18 07:05:43,411 DEBUG [M:0;jenkins-hbase4:46813] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-18 07:05:43,426 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-18 07:05:43,426 INFO [M:0;jenkins-hbase4:46813] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-18 07:05:43,426 INFO [M:0;jenkins-hbase4:46813] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46813 2023-07-18 07:05:43,428 DEBUG [M:0;jenkins-hbase4:46813] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46813,1689663917527 already deleted, retry=false 2023-07-18 07:05:43,602 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 07:05:43,602 INFO [M:0;jenkins-hbase4:46813] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,46813,1689663917527; zookeeper connection closed. 2023-07-18 07:05:43,602 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): master:46813-0x101773fcd8e0000, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 07:05:43,702 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 07:05:43,702 INFO [RS:2;jenkins-hbase4:35743] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,35743,1689663923992; zookeeper connection closed. 2023-07-18 07:05:43,702 DEBUG [Listener at localhost/37223-EventThread] zookeeper.ZKWatcher(600): regionserver:35743-0x101773fcd8e0008, quorum=127.0.0.1:50929, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-18 07:05:43,702 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@35b4783e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@35b4783e 2023-07-18 07:05:43,703 INFO [Listener at localhost/37223] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-18 07:05:43,703 WARN [Listener at localhost/37223] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-18 07:05:43,708 INFO [Listener at localhost/37223] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-18 07:05:43,813 WARN [BP-1725844787-172.31.14.131-1689663914536 heartbeating to localhost/127.0.0.1:37543] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-18 07:05:43,813 WARN [BP-1725844787-172.31.14.131-1689663914536 heartbeating to localhost/127.0.0.1:37543] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1725844787-172.31.14.131-1689663914536 (Datanode Uuid 5fb1d18f-cfa1-4cf8-adb2-38ba3992790e) service to localhost/127.0.0.1:37543 2023-07-18 07:05:43,816 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c/dfs/data/data3/current/BP-1725844787-172.31.14.131-1689663914536] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-18 07:05:43,816 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c/dfs/data/data4/current/BP-1725844787-172.31.14.131-1689663914536] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-18 07:05:43,817 WARN [Listener at localhost/37223] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-18 07:05:43,820 INFO [Listener at localhost/37223] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-18 07:05:43,925 WARN [BP-1725844787-172.31.14.131-1689663914536 heartbeating to localhost/127.0.0.1:37543] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-18 07:05:43,925 WARN [BP-1725844787-172.31.14.131-1689663914536 heartbeating to localhost/127.0.0.1:37543] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1725844787-172.31.14.131-1689663914536 (Datanode Uuid a0460555-2eaf-4d85-9d1d-21d22e72bab6) service to localhost/127.0.0.1:37543 2023-07-18 07:05:43,926 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c/dfs/data/data1/current/BP-1725844787-172.31.14.131-1689663914536] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-18 07:05:43,927 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66ce80e6-6143-de6e-564f-9fa532c78926/cluster_930134e9-fd37-28df-263c-ff123335896c/dfs/data/data2/current/BP-1725844787-172.31.14.131-1689663914536] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-18 07:05:43,966 INFO [Listener at localhost/37223] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-18 07:05:44,064 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-18 07:05:44,079 INFO [Listener at localhost/37223] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-18 07:05:44,116 INFO [Listener at localhost/37223] hbase.HBaseTestingUtility(1293): Minicluster is down