2023-07-12 14:15:32,839 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-12 14:15:33,085 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425 2023-07-12 14:15:33,100 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 14:15:33,101 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79, deleteOnExit=true 2023-07-12 14:15:33,101 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 14:15:33,101 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/test.cache.data in system properties and HBase conf 2023-07-12 14:15:33,102 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 14:15:33,102 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/hadoop.log.dir in system properties and HBase conf 2023-07-12 14:15:33,103 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 14:15:33,103 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 14:15:33,103 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 14:15:33,215 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-12 14:15:33,624 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 14:15:33,628 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 14:15:33,628 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 14:15:33,629 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 14:15:33,629 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 14:15:33,630 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 14:15:33,630 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 14:15:33,630 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 14:15:33,631 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 14:15:33,631 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 14:15:33,632 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/nfs.dump.dir in system properties and HBase conf 2023-07-12 14:15:33,632 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/java.io.tmpdir in system properties and HBase conf 2023-07-12 14:15:33,633 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 14:15:33,633 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 14:15:33,633 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 14:15:34,213 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 14:15:34,216 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 14:15:34,505 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-12 14:15:34,704 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-12 14:15:34,732 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 14:15:34,799 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 14:15:34,855 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/java.io.tmpdir/Jetty_localhost_localdomain_33997_hdfs____.mcympb/webapp 2023-07-12 14:15:35,017 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:33997 2023-07-12 14:15:35,027 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 14:15:35,027 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 14:15:35,519 WARN [Listener at localhost.localdomain/39107] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 14:15:35,652 WARN [Listener at localhost.localdomain/39107] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 14:15:35,670 WARN [Listener at localhost.localdomain/39107] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 14:15:35,676 INFO [Listener at localhost.localdomain/39107] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 14:15:35,681 INFO [Listener at localhost.localdomain/39107] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/java.io.tmpdir/Jetty_localhost_35787_datanode____m7n6pn/webapp 2023-07-12 14:15:35,767 INFO [Listener at localhost.localdomain/39107] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35787 2023-07-12 14:15:36,152 WARN [Listener at localhost.localdomain/34213] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 14:15:36,171 WARN [Listener at localhost.localdomain/34213] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 14:15:36,178 WARN [Listener at localhost.localdomain/34213] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 14:15:36,180 INFO [Listener at localhost.localdomain/34213] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 14:15:36,185 INFO [Listener at localhost.localdomain/34213] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/java.io.tmpdir/Jetty_localhost_36941_datanode____.fvbxpi/webapp 2023-07-12 14:15:36,293 INFO [Listener at localhost.localdomain/34213] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36941 2023-07-12 14:15:36,306 WARN [Listener at localhost.localdomain/36341] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 14:15:36,760 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc139b176e43c8c81: Processing first storage report for DS-ab4ecc90-009e-42aa-b697-0f0860452658 from datanode 5ee61f3b-3d89-4c08-ab5d-f8a4364a1745 2023-07-12 14:15:36,762 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc139b176e43c8c81: from storage DS-ab4ecc90-009e-42aa-b697-0f0860452658 node DatanodeRegistration(127.0.0.1:44245, datanodeUuid=5ee61f3b-3d89-4c08-ab5d-f8a4364a1745, infoPort=35073, infoSecurePort=0, ipcPort=36341, storageInfo=lv=-57;cid=testClusterID;nsid=1087302574;c=1689171334293), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-12 14:15:36,763 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x94e858aa059e9330: Processing first storage report for DS-3c98cb4e-0a14-440b-97e9-281b345f6a36 from datanode b3f92d14-88f4-4586-945a-eecfefe8b150 2023-07-12 14:15:36,763 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x94e858aa059e9330: from storage DS-3c98cb4e-0a14-440b-97e9-281b345f6a36 node DatanodeRegistration(127.0.0.1:42549, datanodeUuid=b3f92d14-88f4-4586-945a-eecfefe8b150, infoPort=44841, infoSecurePort=0, ipcPort=34213, storageInfo=lv=-57;cid=testClusterID;nsid=1087302574;c=1689171334293), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 14:15:36,763 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc139b176e43c8c81: Processing first storage report for DS-f0ee8948-ea7f-46d3-a6c0-198147eae19d from datanode 5ee61f3b-3d89-4c08-ab5d-f8a4364a1745 2023-07-12 14:15:36,763 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc139b176e43c8c81: from storage DS-f0ee8948-ea7f-46d3-a6c0-198147eae19d node DatanodeRegistration(127.0.0.1:44245, datanodeUuid=5ee61f3b-3d89-4c08-ab5d-f8a4364a1745, infoPort=35073, infoSecurePort=0, ipcPort=36341, storageInfo=lv=-57;cid=testClusterID;nsid=1087302574;c=1689171334293), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 14:15:36,763 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x94e858aa059e9330: Processing first storage report for DS-d4f325f5-82b3-4176-b6df-910e3cde360f from datanode b3f92d14-88f4-4586-945a-eecfefe8b150 2023-07-12 14:15:36,764 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x94e858aa059e9330: from storage DS-d4f325f5-82b3-4176-b6df-910e3cde360f node DatanodeRegistration(127.0.0.1:42549, datanodeUuid=b3f92d14-88f4-4586-945a-eecfefe8b150, infoPort=44841, infoSecurePort=0, ipcPort=34213, storageInfo=lv=-57;cid=testClusterID;nsid=1087302574;c=1689171334293), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 14:15:36,865 DEBUG [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425 2023-07-12 14:15:36,930 INFO [Listener at localhost.localdomain/36341] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79/zookeeper_0, clientPort=51038, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 14:15:36,942 INFO [Listener at localhost.localdomain/36341] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51038 2023-07-12 14:15:36,953 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:36,956 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:37,594 INFO [Listener at localhost.localdomain/36341] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8 with version=8 2023-07-12 14:15:37,594 INFO [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/hbase-staging 2023-07-12 14:15:37,893 INFO [Listener at localhost.localdomain/36341] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 14:15:38,372 INFO [Listener at localhost.localdomain/36341] client.ConnectionUtils(127): master/jenkins-hbase17:0 server-side Connection retries=45 2023-07-12 14:15:38,400 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:38,401 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:38,401 INFO [Listener at localhost.localdomain/36341] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 14:15:38,401 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:38,401 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 14:15:38,540 INFO [Listener at localhost.localdomain/36341] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 14:15:38,603 DEBUG [Listener at localhost.localdomain/36341] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-12 14:15:38,688 INFO [Listener at localhost.localdomain/36341] ipc.NettyRpcServer(120): Bind to /136.243.18.41:44183 2023-07-12 14:15:38,697 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:38,700 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:38,724 INFO [Listener at localhost.localdomain/36341] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44183 connecting to ZooKeeper ensemble=127.0.0.1:51038 2023-07-12 14:15:38,761 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:441830x0, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 14:15:38,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:44183-0x10159e299f20000 connected 2023-07-12 14:15:38,791 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 14:15:38,792 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 14:15:38,796 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 14:15:38,803 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44183 2023-07-12 14:15:38,803 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44183 2023-07-12 14:15:38,804 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44183 2023-07-12 14:15:38,805 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44183 2023-07-12 14:15:38,805 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44183 2023-07-12 14:15:38,811 INFO [Listener at localhost.localdomain/36341] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8, hbase.cluster.distributed=false 2023-07-12 14:15:38,874 INFO [Listener at localhost.localdomain/36341] client.ConnectionUtils(127): regionserver/jenkins-hbase17:0 server-side Connection retries=45 2023-07-12 14:15:38,874 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:38,874 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:38,874 INFO [Listener at localhost.localdomain/36341] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 14:15:38,875 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:38,875 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 14:15:38,881 INFO [Listener at localhost.localdomain/36341] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 14:15:38,884 INFO [Listener at localhost.localdomain/36341] ipc.NettyRpcServer(120): Bind to /136.243.18.41:46441 2023-07-12 14:15:38,887 INFO [Listener at localhost.localdomain/36341] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 14:15:38,895 DEBUG [Listener at localhost.localdomain/36341] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 14:15:38,896 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:38,898 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:38,900 INFO [Listener at localhost.localdomain/36341] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46441 connecting to ZooKeeper ensemble=127.0.0.1:51038 2023-07-12 14:15:38,903 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:464410x0, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 14:15:38,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:46441-0x10159e299f20001 connected 2023-07-12 14:15:38,904 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 14:15:38,905 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 14:15:38,906 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 14:15:38,906 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46441 2023-07-12 14:15:38,907 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46441 2023-07-12 14:15:38,907 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46441 2023-07-12 14:15:38,908 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46441 2023-07-12 14:15:38,908 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46441 2023-07-12 14:15:38,924 INFO [Listener at localhost.localdomain/36341] client.ConnectionUtils(127): regionserver/jenkins-hbase17:0 server-side Connection retries=45 2023-07-12 14:15:38,924 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:38,924 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:38,925 INFO [Listener at localhost.localdomain/36341] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 14:15:38,925 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:38,925 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 14:15:38,925 INFO [Listener at localhost.localdomain/36341] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 14:15:38,927 INFO [Listener at localhost.localdomain/36341] ipc.NettyRpcServer(120): Bind to /136.243.18.41:34175 2023-07-12 14:15:38,927 INFO [Listener at localhost.localdomain/36341] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 14:15:38,931 DEBUG [Listener at localhost.localdomain/36341] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 14:15:38,932 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:38,934 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:38,935 INFO [Listener at localhost.localdomain/36341] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34175 connecting to ZooKeeper ensemble=127.0.0.1:51038 2023-07-12 14:15:38,938 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:341750x0, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 14:15:38,939 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): regionserver:341750x0, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 14:15:38,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34175-0x10159e299f20002 connected 2023-07-12 14:15:38,940 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 14:15:38,941 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 14:15:38,942 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34175 2023-07-12 14:15:38,942 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34175 2023-07-12 14:15:38,942 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34175 2023-07-12 14:15:38,943 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34175 2023-07-12 14:15:38,943 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34175 2023-07-12 14:15:38,944 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:15:38,952 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 14:15:38,953 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:15:38,971 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 14:15:38,971 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 14:15:38,971 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 14:15:38,971 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:15:38,972 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 14:15:38,973 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase17.apache.org,44183,1689171337725 from backup master directory 2023-07-12 14:15:38,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 14:15:38,976 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:15:38,976 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 14:15:38,977 WARN [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 14:15:38,977 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:15:38,979 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-12 14:15:38,980 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-12 14:15:39,071 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/hbase.id with ID: 70f49905-7e5f-4cbe-8336-b641ab793488 2023-07-12 14:15:39,117 INFO [master/jenkins-hbase17:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:39,132 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:15:39,171 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x28274dee to 127.0.0.1:51038 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 14:15:39,202 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a9ce088, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 14:15:39,224 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 14:15:39,226 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 14:15:39,244 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-12 14:15:39,245 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-12 14:15:39,248 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 14:15:39,252 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 14:15:39,254 INFO [master/jenkins-hbase17:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 14:15:39,289 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/data/master/store-tmp 2023-07-12 14:15:39,326 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:15:39,326 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 14:15:39,326 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 14:15:39,327 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 14:15:39,327 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 14:15:39,327 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 14:15:39,327 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 14:15:39,327 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 14:15:39,329 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/WALs/jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:15:39,355 INFO [master/jenkins-hbase17:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C44183%2C1689171337725, suffix=, logDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/WALs/jenkins-hbase17.apache.org,44183,1689171337725, archiveDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/oldWALs, maxLogs=10 2023-07-12 14:15:39,417 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK] 2023-07-12 14:15:39,417 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK] 2023-07-12 14:15:39,426 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 14:15:39,523 INFO [master/jenkins-hbase17:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/WALs/jenkins-hbase17.apache.org,44183,1689171337725/jenkins-hbase17.apache.org%2C44183%2C1689171337725.1689171339365 2023-07-12 14:15:39,524 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK], DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK]] 2023-07-12 14:15:39,525 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:15:39,525 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:15:39,529 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 14:15:39,530 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 14:15:39,642 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 14:15:39,652 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 14:15:39,696 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 14:15:39,716 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:39,727 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 14:15:39,730 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 14:15:39,753 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 14:15:39,762 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 14:15:39,763 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72842974, jitterRate=0.08544489741325378}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 14:15:39,764 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 14:15:39,766 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 14:15:39,794 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 14:15:39,794 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 14:15:39,798 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 14:15:39,800 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 14:15:39,841 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 40 msec 2023-07-12 14:15:39,841 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 14:15:39,876 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 14:15:39,886 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 14:15:39,920 INFO [master/jenkins-hbase17:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 14:15:39,931 INFO [master/jenkins-hbase17:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 14:15:39,933 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 14:15:39,940 INFO [master/jenkins-hbase17:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 14:15:39,946 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 14:15:39,968 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:15:39,971 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 14:15:39,972 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 14:15:39,994 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 14:15:40,004 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 14:15:40,004 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 14:15:40,004 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 14:15:40,005 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:15:40,005 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase17.apache.org,44183,1689171337725, sessionid=0x10159e299f20000, setting cluster-up flag (Was=false) 2023-07-12 14:15:40,033 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:15:40,041 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 14:15:40,047 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:15:40,056 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:15:40,062 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 14:15:40,063 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:15:40,067 WARN [master/jenkins-hbase17:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.hbase-snapshot/.tmp 2023-07-12 14:15:40,147 INFO [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(951): ClusterId : 70f49905-7e5f-4cbe-8336-b641ab793488 2023-07-12 14:15:40,156 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(951): ClusterId : 70f49905-7e5f-4cbe-8336-b641ab793488 2023-07-12 14:15:40,167 DEBUG [RS:0;jenkins-hbase17:46441] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 14:15:40,167 DEBUG [RS:1;jenkins-hbase17:34175] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 14:15:40,175 DEBUG [RS:0;jenkins-hbase17:46441] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 14:15:40,175 DEBUG [RS:0;jenkins-hbase17:46441] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 14:15:40,183 DEBUG [RS:1;jenkins-hbase17:34175] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 14:15:40,183 DEBUG [RS:1;jenkins-hbase17:34175] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 14:15:40,186 DEBUG [RS:1;jenkins-hbase17:34175] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 14:15:40,188 DEBUG [RS:1;jenkins-hbase17:34175] zookeeper.ReadOnlyZKClient(139): Connect 0x51d36f56 to 127.0.0.1:51038 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 14:15:40,213 DEBUG [RS:0;jenkins-hbase17:46441] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 14:15:40,214 DEBUG [RS:0;jenkins-hbase17:46441] zookeeper.ReadOnlyZKClient(139): Connect 0x14d131f2 to 127.0.0.1:51038 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 14:15:40,227 DEBUG [RS:1;jenkins-hbase17:34175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f148b1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 14:15:40,228 DEBUG [RS:1;jenkins-hbase17:34175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b2b0426, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase17.apache.org/136.243.18.41:0 2023-07-12 14:15:40,240 DEBUG [RS:0;jenkins-hbase17:46441] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a16dd18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 14:15:40,241 DEBUG [RS:0;jenkins-hbase17:46441] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e295a82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase17.apache.org/136.243.18.41:0 2023-07-12 14:15:40,259 DEBUG [RS:1;jenkins-hbase17:34175] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase17:34175 2023-07-12 14:15:40,264 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 14:15:40,259 DEBUG [RS:0;jenkins-hbase17:46441] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase17:46441 2023-07-12 14:15:40,266 INFO [RS:0;jenkins-hbase17:46441] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 14:15:40,267 INFO [RS:0;jenkins-hbase17:46441] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 14:15:40,267 DEBUG [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 14:15:40,267 INFO [RS:1;jenkins-hbase17:34175] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 14:15:40,268 INFO [RS:1;jenkins-hbase17:34175] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 14:15:40,268 DEBUG [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 14:15:40,271 INFO [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase17.apache.org,44183,1689171337725 with isa=jenkins-hbase17.apache.org/136.243.18.41:46441, startcode=1689171338873 2023-07-12 14:15:40,272 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase17.apache.org,44183,1689171337725 with isa=jenkins-hbase17.apache.org/136.243.18.41:34175, startcode=1689171338924 2023-07-12 14:15:40,281 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-07-12 14:15:40,281 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-07-12 14:15:40,284 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-07-12 14:15:40,285 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-07-12 14:15:40,285 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase17:0, corePoolSize=10, maxPoolSize=10 2023-07-12 14:15:40,285 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,285 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=2, maxPoolSize=2 2023-07-12 14:15:40,286 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,308 DEBUG [RS:1;jenkins-hbase17:34175] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 14:15:40,309 DEBUG [RS:0;jenkins-hbase17:46441] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 14:15:40,312 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689171370312 2023-07-12 14:15:40,315 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 14:15:40,333 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 14:15:40,335 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 14:15:40,345 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 14:15:40,355 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 14:15:40,356 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 14:15:40,356 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 14:15:40,356 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 14:15:40,358 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 14:15:40,363 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,388 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 14:15:40,392 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 14:15:40,393 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 14:15:40,417 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 14:15:40,417 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 14:15:40,426 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.large.0-1689171340419,5,FailOnTimeoutGroup] 2023-07-12 14:15:40,433 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:56123, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 14:15:40,437 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.small.0-1689171340426,5,FailOnTimeoutGroup] 2023-07-12 14:15:40,440 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,433 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:46305, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 14:15:40,455 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 14:15:40,459 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,459 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,471 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44183] master.ServerManager(394): Registering regionserver=jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:40,483 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44183] master.ServerManager(394): Registering regionserver=jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:40,502 DEBUG [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8 2023-07-12 14:15:40,502 DEBUG [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:39107 2023-07-12 14:15:40,502 DEBUG [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 14:15:40,502 DEBUG [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8 2023-07-12 14:15:40,505 DEBUG [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:39107 2023-07-12 14:15:40,505 DEBUG [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 14:15:40,512 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:15:40,515 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 14:15:40,516 DEBUG [RS:1;jenkins-hbase17:34175] zookeeper.ZKUtil(162): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:40,516 WARN [RS:1;jenkins-hbase17:34175] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 14:15:40,518 INFO [RS:1;jenkins-hbase17:34175] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 14:15:40,518 DEBUG [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:40,520 DEBUG [RS:0;jenkins-hbase17:46441] zookeeper.ZKUtil(162): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:40,520 WARN [RS:0;jenkins-hbase17:46441] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 14:15:40,520 INFO [RS:0;jenkins-hbase17:46441] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 14:15:40,521 DEBUG [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:40,522 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 14:15:40,522 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8 2023-07-12 14:15:40,523 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase17.apache.org,34175,1689171338924] 2023-07-12 14:15:40,526 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase17.apache.org,46441,1689171338873] 2023-07-12 14:15:40,540 DEBUG [RS:1;jenkins-hbase17:34175] zookeeper.ZKUtil(162): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:40,540 DEBUG [RS:0;jenkins-hbase17:46441] zookeeper.ZKUtil(162): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:40,543 DEBUG [RS:0;jenkins-hbase17:46441] zookeeper.ZKUtil(162): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:40,543 DEBUG [RS:1;jenkins-hbase17:34175] zookeeper.ZKUtil(162): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:40,558 DEBUG [RS:1;jenkins-hbase17:34175] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 14:15:40,565 DEBUG [RS:0;jenkins-hbase17:46441] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 14:15:40,575 INFO [RS:1;jenkins-hbase17:34175] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 14:15:40,575 INFO [RS:0;jenkins-hbase17:46441] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 14:15:40,595 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:15:40,606 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 14:15:40,610 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/info 2023-07-12 14:15:40,611 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 14:15:40,614 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:40,614 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 14:15:40,619 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/rep_barrier 2023-07-12 14:15:40,621 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 14:15:40,622 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:40,623 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 14:15:40,627 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/table 2023-07-12 14:15:40,627 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 14:15:40,629 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:40,638 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740 2023-07-12 14:15:40,640 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740 2023-07-12 14:15:40,645 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 14:15:40,651 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 14:15:40,660 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 14:15:40,661 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63651827, jitterRate=-0.05151386559009552}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 14:15:40,662 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 14:15:40,662 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 14:15:40,662 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 14:15:40,662 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 14:15:40,662 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 14:15:40,662 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 14:15:40,663 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 14:15:40,665 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 14:15:40,667 INFO [RS:1;jenkins-hbase17:34175] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 14:15:40,671 INFO [RS:0;jenkins-hbase17:46441] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 14:15:40,680 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 14:15:40,680 INFO [RS:1;jenkins-hbase17:34175] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 14:15:40,681 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 14:15:40,681 INFO [RS:1;jenkins-hbase17:34175] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,680 INFO [RS:0;jenkins-hbase17:46441] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 14:15:40,681 INFO [RS:0;jenkins-hbase17:46441] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,685 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 14:15:40,688 INFO [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 14:15:40,700 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 14:15:40,700 INFO [RS:0;jenkins-hbase17:46441] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,700 INFO [RS:1;jenkins-hbase17:34175] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,700 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,701 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,701 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,701 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,701 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,701 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,701 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0, corePoolSize=2, maxPoolSize=2 2023-07-12 14:15:40,702 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0, corePoolSize=2, maxPoolSize=2 2023-07-12 14:15:40,702 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:0;jenkins-hbase17:46441] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,702 DEBUG [RS:1;jenkins-hbase17:34175] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:40,704 INFO [RS:0;jenkins-hbase17:46441] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,706 INFO [RS:1;jenkins-hbase17:34175] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,706 INFO [RS:0;jenkins-hbase17:46441] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,706 INFO [RS:1;jenkins-hbase17:34175] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,706 INFO [RS:0;jenkins-hbase17:46441] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,706 INFO [RS:1;jenkins-hbase17:34175] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,715 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 14:15:40,719 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 14:15:40,727 INFO [RS:0;jenkins-hbase17:46441] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 14:15:40,727 INFO [RS:1;jenkins-hbase17:34175] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 14:15:40,730 INFO [RS:0;jenkins-hbase17:46441] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,46441,1689171338873-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,730 INFO [RS:1;jenkins-hbase17:34175] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,34175,1689171338924-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:40,751 INFO [RS:0;jenkins-hbase17:46441] regionserver.Replication(203): jenkins-hbase17.apache.org,46441,1689171338873 started 2023-07-12 14:15:40,751 INFO [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1637): Serving as jenkins-hbase17.apache.org,46441,1689171338873, RpcServer on jenkins-hbase17.apache.org/136.243.18.41:46441, sessionid=0x10159e299f20001 2023-07-12 14:15:40,768 INFO [RS:1;jenkins-hbase17:34175] regionserver.Replication(203): jenkins-hbase17.apache.org,34175,1689171338924 started 2023-07-12 14:15:40,768 DEBUG [RS:0;jenkins-hbase17:46441] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 14:15:40,768 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1637): Serving as jenkins-hbase17.apache.org,34175,1689171338924, RpcServer on jenkins-hbase17.apache.org/136.243.18.41:34175, sessionid=0x10159e299f20002 2023-07-12 14:15:40,768 DEBUG [RS:0;jenkins-hbase17:46441] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:40,769 DEBUG [RS:1;jenkins-hbase17:34175] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 14:15:40,769 DEBUG [RS:0;jenkins-hbase17:46441] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,46441,1689171338873' 2023-07-12 14:15:40,769 DEBUG [RS:1;jenkins-hbase17:34175] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:40,773 DEBUG [RS:1;jenkins-hbase17:34175] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,34175,1689171338924' 2023-07-12 14:15:40,773 DEBUG [RS:0;jenkins-hbase17:46441] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 14:15:40,773 DEBUG [RS:1;jenkins-hbase17:34175] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 14:15:40,774 DEBUG [RS:0;jenkins-hbase17:46441] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 14:15:40,774 DEBUG [RS:1;jenkins-hbase17:34175] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 14:15:40,775 DEBUG [RS:1;jenkins-hbase17:34175] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 14:15:40,775 DEBUG [RS:0;jenkins-hbase17:46441] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 14:15:40,775 DEBUG [RS:1;jenkins-hbase17:34175] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 14:15:40,775 DEBUG [RS:0;jenkins-hbase17:46441] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 14:15:40,775 DEBUG [RS:1;jenkins-hbase17:34175] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:40,775 DEBUG [RS:0;jenkins-hbase17:46441] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:40,776 DEBUG [RS:1;jenkins-hbase17:34175] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,34175,1689171338924' 2023-07-12 14:15:40,776 DEBUG [RS:1;jenkins-hbase17:34175] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 14:15:40,776 DEBUG [RS:0;jenkins-hbase17:46441] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,46441,1689171338873' 2023-07-12 14:15:40,776 DEBUG [RS:0;jenkins-hbase17:46441] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 14:15:40,777 DEBUG [RS:1;jenkins-hbase17:34175] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 14:15:40,777 DEBUG [RS:0;jenkins-hbase17:46441] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 14:15:40,777 DEBUG [RS:1;jenkins-hbase17:34175] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 14:15:40,777 DEBUG [RS:0;jenkins-hbase17:46441] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 14:15:40,777 INFO [RS:1;jenkins-hbase17:34175] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 14:15:40,777 INFO [RS:0;jenkins-hbase17:46441] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 14:15:40,778 INFO [RS:1;jenkins-hbase17:34175] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 14:15:40,778 INFO [RS:0;jenkins-hbase17:46441] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 14:15:40,872 DEBUG [jenkins-hbase17:44183] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 14:15:40,876 DEBUG [jenkins-hbase17:44183] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase17.apache.org=0} racks are {/default-rack=0} 2023-07-12 14:15:40,884 DEBUG [jenkins-hbase17:44183] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 14:15:40,884 DEBUG [jenkins-hbase17:44183] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 14:15:40,884 DEBUG [jenkins-hbase17:44183] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 14:15:40,887 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase17.apache.org,34175,1689171338924, state=OPENING 2023-07-12 14:15:40,896 INFO [RS:0;jenkins-hbase17:46441] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C46441%2C1689171338873, suffix=, logDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873, archiveDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs, maxLogs=32 2023-07-12 14:15:40,896 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 14:15:40,897 INFO [RS:1;jenkins-hbase17:34175] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C34175%2C1689171338924, suffix=, logDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924, archiveDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs, maxLogs=32 2023-07-12 14:15:40,898 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:15:40,899 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 14:15:40,903 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase17.apache.org,34175,1689171338924}] 2023-07-12 14:15:40,929 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK] 2023-07-12 14:15:40,930 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK] 2023-07-12 14:15:40,930 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK] 2023-07-12 14:15:40,930 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK] 2023-07-12 14:15:40,946 INFO [RS:0;jenkins-hbase17:46441] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899 2023-07-12 14:15:40,946 INFO [RS:1;jenkins-hbase17:34175] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900 2023-07-12 14:15:40,946 DEBUG [RS:0;jenkins-hbase17:46441] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK], DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK]] 2023-07-12 14:15:40,946 DEBUG [RS:1;jenkins-hbase17:34175] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK], DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK]] 2023-07-12 14:15:41,108 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:41,111 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 14:15:41,115 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:56250, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 14:15:41,128 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 14:15:41,129 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 14:15:41,133 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924, archiveDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs, maxLogs=32 2023-07-12 14:15:41,154 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK] 2023-07-12 14:15:41,154 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK] 2023-07-12 14:15:41,164 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta 2023-07-12 14:15:41,165 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK], DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK]] 2023-07-12 14:15:41,166 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:15:41,168 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 14:15:41,190 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 14:15:41,198 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 14:15:41,206 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 14:15:41,207 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:15:41,207 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 14:15:41,207 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 14:15:41,212 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 14:15:41,226 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/info 2023-07-12 14:15:41,228 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/info 2023-07-12 14:15:41,229 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 14:15:41,231 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:41,232 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 14:15:41,234 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/rep_barrier 2023-07-12 14:15:41,234 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/rep_barrier 2023-07-12 14:15:41,235 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 14:15:41,236 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:41,237 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 14:15:41,240 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/table 2023-07-12 14:15:41,240 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/table 2023-07-12 14:15:41,240 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 14:15:41,242 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:41,245 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740 2023-07-12 14:15:41,249 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740 2023-07-12 14:15:41,253 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 14:15:41,257 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 14:15:41,261 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75269723, jitterRate=0.12160627543926239}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 14:15:41,261 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 14:15:41,281 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689171341100 2023-07-12 14:15:41,311 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 14:15:41,312 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 14:15:41,312 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase17.apache.org,34175,1689171338924, state=OPEN 2023-07-12 14:15:41,315 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 14:15:41,315 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 14:15:41,320 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 14:15:41,321 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase17.apache.org,34175,1689171338924 in 412 msec 2023-07-12 14:15:41,328 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 14:15:41,329 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 623 msec 2023-07-12 14:15:41,336 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1900 sec 2023-07-12 14:15:41,336 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689171341336, completionTime=-1 2023-07-12 14:15:41,336 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 14:15:41,337 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 14:15:41,441 DEBUG [hconnection-0x4fd24090-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 14:15:41,444 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:56252, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 14:15:41,489 INFO [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 14:15:41,489 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689171401489 2023-07-12 14:15:41,489 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689171461489 2023-07-12 14:15:41,489 INFO [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 151 msec 2023-07-12 14:15:41,519 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44183,1689171337725-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:41,520 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44183,1689171337725-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:41,520 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44183,1689171337725-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:41,522 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase17:44183, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:41,523 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:41,531 DEBUG [master/jenkins-hbase17:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 14:15:41,541 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 14:15:41,544 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 14:15:41,556 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 14:15:41,559 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 14:15:41,562 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 14:15:41,612 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:15:41,627 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0 empty. 2023-07-12 14:15:41,628 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:15:41,628 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 14:15:41,667 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 14:15:41,670 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => c0e52329bdbaee97aeea58f1b40da9f0, NAME => 'hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp 2023-07-12 14:15:41,695 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:15:41,695 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing c0e52329bdbaee97aeea58f1b40da9f0, disabling compactions & flushes 2023-07-12 14:15:41,695 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:15:41,695 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:15:41,695 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. after waiting 0 ms 2023-07-12 14:15:41,696 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:15:41,697 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:15:41,697 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for c0e52329bdbaee97aeea58f1b40da9f0: 2023-07-12 14:15:41,703 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 14:15:41,721 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689171341707"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689171341707"}]},"ts":"1689171341707"} 2023-07-12 14:15:41,759 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 14:15:41,761 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 14:15:41,766 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689171341762"}]},"ts":"1689171341762"} 2023-07-12 14:15:41,772 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 14:15:41,775 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase17.apache.org=0} racks are {/default-rack=0} 2023-07-12 14:15:41,778 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 14:15:41,778 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 14:15:41,778 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 14:15:41,781 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0e52329bdbaee97aeea58f1b40da9f0, ASSIGN}] 2023-07-12 14:15:41,785 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0e52329bdbaee97aeea58f1b40da9f0, ASSIGN 2023-07-12 14:15:41,787 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c0e52329bdbaee97aeea58f1b40da9f0, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,34175,1689171338924; forceNewPlan=false, retain=false 2023-07-12 14:15:41,941 INFO [jenkins-hbase17:44183] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 14:15:41,943 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c0e52329bdbaee97aeea58f1b40da9f0, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:41,943 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689171341942"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689171341942"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689171341942"}]},"ts":"1689171341942"} 2023-07-12 14:15:41,948 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c0e52329bdbaee97aeea58f1b40da9f0, server=jenkins-hbase17.apache.org,34175,1689171338924}] 2023-07-12 14:15:42,117 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:15:42,119 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c0e52329bdbaee97aeea58f1b40da9f0, NAME => 'hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:15:42,120 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:15:42,120 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:15:42,121 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:15:42,121 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:15:42,128 INFO [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:15:42,132 DEBUG [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/info 2023-07-12 14:15:42,132 DEBUG [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/info 2023-07-12 14:15:42,133 INFO [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c0e52329bdbaee97aeea58f1b40da9f0 columnFamilyName info 2023-07-12 14:15:42,135 INFO [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] regionserver.HStore(310): Store=c0e52329bdbaee97aeea58f1b40da9f0/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:42,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:15:42,142 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:15:42,155 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:15:42,165 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 14:15:42,167 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened c0e52329bdbaee97aeea58f1b40da9f0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65134038, jitterRate=-0.02942720055580139}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 14:15:42,167 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for c0e52329bdbaee97aeea58f1b40da9f0: 2023-07-12 14:15:42,171 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0., pid=6, masterSystemTime=1689171342104 2023-07-12 14:15:42,177 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:15:42,178 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:15:42,179 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c0e52329bdbaee97aeea58f1b40da9f0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:42,180 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689171342178"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689171342178"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689171342178"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689171342178"}]},"ts":"1689171342178"} 2023-07-12 14:15:42,191 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 14:15:42,191 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c0e52329bdbaee97aeea58f1b40da9f0, server=jenkins-hbase17.apache.org,34175,1689171338924 in 237 msec 2023-07-12 14:15:42,196 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 14:15:42,197 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c0e52329bdbaee97aeea58f1b40da9f0, ASSIGN in 410 msec 2023-07-12 14:15:42,198 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 14:15:42,199 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689171342198"}]},"ts":"1689171342198"} 2023-07-12 14:15:42,207 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 14:15:42,213 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 14:15:42,219 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 668 msec 2023-07-12 14:15:42,260 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 14:15:42,261 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 14:15:42,261 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:15:42,343 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 14:15:42,360 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 14:15:42,368 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 37 msec 2023-07-12 14:15:42,380 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 14:15:42,396 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 14:15:42,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 21 msec 2023-07-12 14:15:42,417 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 14:15:42,419 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 14:15:42,421 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.444sec 2023-07-12 14:15:42,424 INFO [master/jenkins-hbase17:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 14:15:42,426 INFO [master/jenkins-hbase17:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 14:15:42,426 INFO [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 14:15:42,428 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44183,1689171337725-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 14:15:42,430 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44183,1689171337725-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 14:15:42,446 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 14:15:42,462 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ReadOnlyZKClient(139): Connect 0x43f27a6f to 127.0.0.1:51038 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 14:15:42,472 DEBUG [Listener at localhost.localdomain/36341] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4918c55a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 14:15:42,490 DEBUG [hconnection-0x74f4ba83-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 14:15:42,508 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:56260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 14:15:42,519 INFO [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:15:42,532 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 14:15:42,539 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 14:15:42,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 14:15:42,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-12 14:15:42,563 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 14:15:42,566 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 14:15:42,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] master.MasterRpcServices(700): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-12 14:15:42,570 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp/data/default/t/d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:42,572 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp/data/default/t/d808f9db952c342c04ce9909916ad2b0 empty. 2023-07-12 14:15:42,575 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp/data/default/t/d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:42,575 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-12 14:15:42,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 14:15:42,611 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-12 14:15:42,613 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => d808f9db952c342c04ce9909916ad2b0, NAME => 't,,1689171342552.d808f9db952c342c04ce9909916ad2b0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/.tmp 2023-07-12 14:15:42,681 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:15:42,681 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing d808f9db952c342c04ce9909916ad2b0, disabling compactions & flushes 2023-07-12 14:15:42,681 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:42,681 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:42,681 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. after waiting 0 ms 2023-07-12 14:15:42,681 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:42,681 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:42,682 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for d808f9db952c342c04ce9909916ad2b0: 2023-07-12 14:15:42,691 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 14:15:42,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 14:15:42,695 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689171342694"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689171342694"}]},"ts":"1689171342694"} 2023-07-12 14:15:42,699 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 14:15:42,701 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 14:15:42,702 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689171342702"}]},"ts":"1689171342702"} 2023-07-12 14:15:42,717 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-12 14:15:42,722 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase17.apache.org=0} racks are {/default-rack=0} 2023-07-12 14:15:42,723 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 14:15:42,723 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 14:15:42,723 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 14:15:42,724 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN}] 2023-07-12 14:15:42,730 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN 2023-07-12 14:15:42,734 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,46441,1689171338873; forceNewPlan=false, retain=false 2023-07-12 14:15:42,884 INFO [jenkins-hbase17:44183] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 14:15:42,885 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d808f9db952c342c04ce9909916ad2b0, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:42,886 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689171342885"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689171342885"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689171342885"}]},"ts":"1689171342885"} 2023-07-12 14:15:42,890 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure d808f9db952c342c04ce9909916ad2b0, server=jenkins-hbase17.apache.org,46441,1689171338873}] 2023-07-12 14:15:42,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 14:15:43,045 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:43,046 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 14:15:43,049 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 14:15:43,057 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(130): Open t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d808f9db952c342c04ce9909916ad2b0, NAME => 't,,1689171342552.d808f9db952c342c04ce9909916ad2b0.', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:15:43,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:43,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:15:43,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:43,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:43,064 INFO [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:43,067 DEBUG [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/f 2023-07-12 14:15:43,067 DEBUG [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/f 2023-07-12 14:15:43,068 INFO [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d808f9db952c342c04ce9909916ad2b0 columnFamilyName f 2023-07-12 14:15:43,069 INFO [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] regionserver.HStore(310): Store=d808f9db952c342c04ce9909916ad2b0/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:43,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:43,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:43,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:43,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 14:15:43,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened d808f9db952c342c04ce9909916ad2b0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63726835, jitterRate=-0.050396159291267395}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 14:15:43,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for d808f9db952c342c04ce9909916ad2b0: 2023-07-12 14:15:43,089 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689171342552.d808f9db952c342c04ce9909916ad2b0., pid=11, masterSystemTime=1689171343045 2023-07-12 14:15:43,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,095 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(158): Opened t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,096 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d808f9db952c342c04ce9909916ad2b0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:43,096 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689171343096"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689171343096"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689171343096"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689171343096"}]},"ts":"1689171343096"} 2023-07-12 14:15:43,105 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 14:15:43,105 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure d808f9db952c342c04ce9909916ad2b0, server=jenkins-hbase17.apache.org,46441,1689171338873 in 210 msec 2023-07-12 14:15:43,109 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 14:15:43,110 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN in 381 msec 2023-07-12 14:15:43,111 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 14:15:43,112 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689171343111"}]},"ts":"1689171343111"} 2023-07-12 14:15:43,116 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-12 14:15:43,119 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 14:15:43,123 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 562 msec 2023-07-12 14:15:43,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 14:15:43,201 INFO [Listener at localhost.localdomain/36341] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-12 14:15:43,201 DEBUG [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-12 14:15:43,203 INFO [Listener at localhost.localdomain/36341] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 14:15:43,209 INFO [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-12 14:15:43,209 INFO [Listener at localhost.localdomain/36341] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 14:15:43,210 INFO [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-12 14:15:43,220 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 14:15:43,225 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 14:15:43,289 INFO [Listener at localhost.localdomain/36341] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=293, OpenFileDescriptor=580, MaxFileDescriptor=60000, SystemLoadAverage=473, ProcessCount=175, AvailableMemoryMB=2600 2023-07-12 14:15:43,291 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-12 14:15:43,339 DEBUG [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-12 14:15:43,342 DEBUG [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,342 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-12 14:15:43,342 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-12 14:15:43,361 INFO [Listener at localhost.localdomain/36341] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689171342198/Put/vlen=2/seqid=0} 2023-07-12 14:15:43,361 INFO [Listener at localhost.localdomain/36341] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-12 14:15:43,361 INFO [Listener at localhost.localdomain/36341] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0./info:regioninfo/1689171342178/Put/vlen=41/seqid=0, hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0./info:seqnumDuringOpen/1689171342178/Put/vlen=8/seqid=0, hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0./info:server/1689171342178/Put/vlen=32/seqid=0, hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0./info:serverstartcode/1689171342178/Put/vlen=8/seqid=0, hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0./info:sn/1689171341942/Put/vlen=46/seqid=0, hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0./info:state/1689171342178/Put/vlen=4/seqid=0} 2023-07-12 14:15:43,362 INFO [Listener at localhost.localdomain/36341] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => c0e52329bdbaee97aeea58f1b40da9f0, NAME => 'hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:15:43,362 INFO [Listener at localhost.localdomain/36341] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689171343111/Put/vlen=2/seqid=0} 2023-07-12 14:15:43,363 INFO [Listener at localhost.localdomain/36341] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-12 14:15:43,363 INFO [Listener at localhost.localdomain/36341] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689171342552.d808f9db952c342c04ce9909916ad2b0./info:regioninfo/1689171343096/Put/vlen=35/seqid=0, t,,1689171342552.d808f9db952c342c04ce9909916ad2b0./info:seqnumDuringOpen/1689171343096/Put/vlen=8/seqid=0, t,,1689171342552.d808f9db952c342c04ce9909916ad2b0./info:server/1689171343096/Put/vlen=32/seqid=0, t,,1689171342552.d808f9db952c342c04ce9909916ad2b0./info:serverstartcode/1689171343096/Put/vlen=8/seqid=0, t,,1689171342552.d808f9db952c342c04ce9909916ad2b0./info:sn/1689171342885/Put/vlen=46/seqid=0, t,,1689171342552.d808f9db952c342c04ce9909916ad2b0./info:state/1689171343096/Put/vlen=4/seqid=0} 2023-07-12 14:15:43,363 INFO [Listener at localhost.localdomain/36341] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => d808f9db952c342c04ce9909916ad2b0, NAME => 't,,1689171342552.d808f9db952c342c04ce9909916ad2b0.', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:15:43,364 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ReadOnlyZKClient(139): Connect 0x7f6eb20c to 127.0.0.1:51038 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 14:15:43,380 DEBUG [Listener at localhost.localdomain/36341] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66a2526e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 14:15:43,380 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-12 14:15:43,380 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-12 14:15:43,384 DEBUG [hconnection-0x288f21dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 14:15:43,392 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:56270, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 14:15:43,400 DEBUG [hconnection-0x288f21dc-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 14:15:43,414 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37446, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 14:15:43,416 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-12 14:15:43,420 ERROR [Listener at localhost.localdomain/36341] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase17.apache.org,46441,1689171338873: die! ***** 2023-07-12 14:15:43,421 ERROR [Listener at localhost.localdomain/36341] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 14:15:43,424 DEBUG [Listener at localhost.localdomain/36341] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 14:15:43,428 DEBUG [Listener at localhost.localdomain/36341] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 14:15:43,428 DEBUG [Listener at localhost.localdomain/36341] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 14:15:43,429 DEBUG [Listener at localhost.localdomain/36341] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 14:15:43,430 INFO [Listener at localhost.localdomain/36341] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1005584384, "init": 524288000, "max": 2051014656, "used": 148186760 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 110898912 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase17.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase17.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase17.apache.org" } ] } 2023-07-12 14:15:43,434 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44183] master.MasterRpcServices(612): jenkins-hbase17.apache.org,46441,1689171338873 reported a fatal error: ***** ABORTING region server jenkins-hbase17.apache.org,46441,1689171338873: die! ***** 2023-07-12 14:15:43,437 INFO [Listener at localhost.localdomain/36341] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase17.apache.org,46441,1689171338873' ***** 2023-07-12 14:15:43,438 INFO [Listener at localhost.localdomain/36341] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-12 14:15:43,439 INFO [RS:0;jenkins-hbase17:46441] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 14:15:43,439 INFO [RS:0;jenkins-hbase17:46441] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 14:15:43,439 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 14:15:43,439 INFO [RS:0;jenkins-hbase17:46441] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 14:15:43,439 INFO [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(3305): Received CLOSE for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:43,445 INFO [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1141): aborting server jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:43,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing d808f9db952c342c04ce9909916ad2b0, disabling compactions & flushes 2023-07-12 14:15:43,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. after waiting 0 ms 2023-07-12 14:15:43,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,447 DEBUG [RS:0;jenkins-hbase17:46441] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14d131f2 to 127.0.0.1:51038 2023-07-12 14:15:43,447 DEBUG [RS:0;jenkins-hbase17:46441] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:15:43,447 INFO [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 14:15:43,447 DEBUG [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1478): Online Regions={d808f9db952c342c04ce9909916ad2b0=t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.} 2023-07-12 14:15:43,447 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,449 DEBUG [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1504): Waiting on d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:43,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46441] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 7989704829255548929 2023-07-12 14:15:43,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for d808f9db952c342c04ce9909916ad2b0: 2023-07-12 14:15:43,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46441] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,46441,1689171338873 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 14:15:43,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:43,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46441] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 136.243.18.41:37446 deadline: 1689171403447, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,46441,1689171338873 aborting 2023-07-12 14:15:43,472 DEBUG [Listener at localhost.localdomain/36341] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,46441,1689171338873 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,46441,1689171338873 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-12 14:15:43,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46441] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,46441,1689171338873 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 14:15:43,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46441] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 136.243.18.41:37446 deadline: 1689171403475, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,46441,1689171338873 aborting 2023-07-12 14:15:43,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46441] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,46441,1689171338873 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 14:15:43,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46441] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 136.243.18.41:37446 deadline: 1689171403590, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,46441,1689171338873 aborting 2023-07-12 14:15:43,650 INFO [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1170): stopping server jenkins-hbase17.apache.org,46441,1689171338873; all regions closed. 2023-07-12 14:15:43,665 DEBUG [RS:0;jenkins-hbase17:46441] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:15:43,666 INFO [RS:0;jenkins-hbase17:46441] regionserver.LeaseManager(133): Closed leases 2023-07-12 14:15:43,666 INFO [RS:0;jenkins-hbase17:46441] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase17:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 14:15:43,666 INFO [RS:0;jenkins-hbase17:46441] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 14:15:43,666 INFO [regionserver/jenkins-hbase17:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 14:15:43,666 INFO [RS:0;jenkins-hbase17:46441] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 14:15:43,667 INFO [RS:0;jenkins-hbase17:46441] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 14:15:43,668 INFO [RS:0;jenkins-hbase17:46441] ipc.NettyRpcServer(158): Stopping server on /136.243.18.41:46441 2023-07-12 14:15:43,680 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:43,680 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:15:43,680 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:43,680 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:15:43,680 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:15:43,681 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase17.apache.org,46441,1689171338873] 2023-07-12 14:15:43,681 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase17.apache.org,46441,1689171338873; numProcessing=1 2023-07-12 14:15:43,683 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase17.apache.org,46441,1689171338873 already deleted, retry=false 2023-07-12 14:15:43,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:43,683 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase17.apache.org,46441,1689171338873 on jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:15:43,684 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase17.apache.org,46441,1689171338873 znode expired, triggering replicatorRemoved event 2023-07-12 14:15:43,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:43,693 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase17.apache.org,46441,1689171338873, splitWal=true, meta=false 2023-07-12 14:15:43,693 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase17.apache.org,46441,1689171338873 (carryingMeta=false) jenkins-hbase17.apache.org,46441,1689171338873/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@13b0e5c5[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-12 14:15:43,697 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase17.apache.org,46441,1689171338873, splitWal=true, meta=false 2023-07-12 14:15:43,699 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase17.apache.org,46441,1689171338873 had 1 regions 2023-07-12 14:15:43,701 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase17.apache.org,46441,1689171338873, splitWal=true, meta=false, isMeta: false 2023-07-12 14:15:43,703 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting 2023-07-12 14:15:43,705 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase17.apache.org,46441,1689171338873 WAL count=1, meta=false 2023-07-12 14:15:43,709 INFO [regionserver/jenkins-hbase17:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 14:15:43,709 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899}] 2023-07-12 14:15:43,722 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:43,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899, worker=jenkins-hbase17.apache.org,34175,1689171338924}] 2023-07-12 14:15:43,783 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 14:15:43,784 INFO [RS:0;jenkins-hbase17:46441] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase17.apache.org,46441,1689171338873; zookeeper connection closed. 2023-07-12 14:15:43,784 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:46441-0x10159e299f20001, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 14:15:43,784 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7ce3c9ab] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7ce3c9ab 2023-07-12 14:15:43,800 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase17.apache.org/136.243.18.41:46441 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase17.apache.org/136.243.18.41:46441 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 14:15:43,801 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase17.apache.org/136.243.18.41:46441 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase17.apache.org/136.243.18.41:46441 2023-07-12 14:15:43,895 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34175] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-12 14:15:43,923 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899, size=1.1 K (1149bytes) 2023-07-12 14:15:43,924 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899 2023-07-12 14:15:43,930 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899 after 6ms 2023-07-12 14:15:43,937 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899 took 16ms 2023-07-12 14:15:43,955 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(346): Last flushed sequenceid for d808f9db952c342c04ce9909916ad2b0: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-12 14:15:43,959 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899 so closing down 2023-07-12 14:15:43,959 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 14:15:43,968 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0-Writer-1] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-12 14:15:43,972 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899.temp 2023-07-12 14:15:43,973 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 14:15:43,992 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-12 14:15:43,999 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899.temp to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000013 2023-07-12 14:15:43,999 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 61 ms; skipped=1; WAL=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899, size=1.1 K, length=1149, corrupted=false, cancelled=false 2023-07-12 14:15:44,000 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899, journal: Splitting hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899, size=1.1 K (1149bytes) at 1689171343923Finishing writing output for hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899 so closing down at 1689171343959 (+36 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899.temp at 1689171343972 (+13 ms)3 split writer threads finished at 1689171343973 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689171343992 (+19 ms)Rename recovered edits hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899.temp to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000013 at 1689171343999 (+7 ms)Processed 11 edits across 1 Regions in 61 ms; skipped=1; WAL=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899, size=1.1 K, length=1149, corrupted=false, cancelled=false at 1689171344000 (+1 ms) 2023-07-12 14:15:44,001 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899 2023-07-12 14:15:44,003 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-12 14:15:44,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] master.HMaster(3641): Remote procedure done, pid=14 2023-07-12 14:15:44,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting/jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899 to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs 2023-07-12 14:15:44,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-12 14:15:44,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899, worker=jenkins-hbase17.apache.org,34175,1689171338924 in 291 msec 2023-07-12 14:15:44,021 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:44,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-12 14:15:44,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase17.apache.org%2C46441%2C1689171338873.1689171340899, worker=jenkins-hbase17.apache.org,34175,1689171338924 in 315 msec 2023-07-12 14:15:44,030 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting dir is empty, no logs to split. 2023-07-12 14:15:44,030 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase17.apache.org,46441,1689171338873 WAL count=0, meta=false 2023-07-12 14:15:44,030 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase17.apache.org,46441,1689171338873 WAL splitting is done? wals=0, meta=false 2023-07-12 14:15:44,035 INFO [PEWorker-5] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase17.apache.org,46441,1689171338873 failed, ignore...File hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,46441,1689171338873-splitting does not exist. 2023-07-12 14:15:44,038 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN}] 2023-07-12 14:15:44,040 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN 2023-07-12 14:15:44,042 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 14:15:44,110 DEBUG [hconnection-0x288f21dc-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase17.apache.org/136.243.18.41:46441 this server is in the failed servers list 2023-07-12 14:15:44,192 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=d808f9db952c342c04ce9909916ad2b0, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:44,193 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689171344192"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689171344192"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689171344192"}]},"ts":"1689171344192"} 2023-07-12 14:15:44,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure d808f9db952c342c04ce9909916ad2b0, server=jenkins-hbase17.apache.org,34175,1689171338924}] 2023-07-12 14:15:44,356 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(130): Open t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:44,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d808f9db952c342c04ce9909916ad2b0, NAME => 't,,1689171342552.d808f9db952c342c04ce9909916ad2b0.', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:15:44,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:44,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:15:44,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:44,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:44,360 INFO [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:44,361 DEBUG [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/f 2023-07-12 14:15:44,361 DEBUG [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/f 2023-07-12 14:15:44,362 INFO [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d808f9db952c342c04ce9909916ad2b0 columnFamilyName f 2023-07-12 14:15:44,363 INFO [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] regionserver.HStore(310): Store=d808f9db952c342c04ce9909916ad2b0/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:15:44,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:44,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:44,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000013 2023-07-12 14:15:44,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000013 2023-07-12 14:15:44,380 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing d808f9db952c342c04ce9909916ad2b0 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-12 14:15:44,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/.tmp/f/1b2b27f8f72d4aa582592210013cd653 2023-07-12 14:15:44,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/.tmp/f/1b2b27f8f72d4aa582592210013cd653 as hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/f/1b2b27f8f72d4aa582592210013cd653 2023-07-12 14:15:44,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/f/1b2b27f8f72d4aa582592210013cd653, entries=10, sequenceid=13, filesize=4.8 K 2023-07-12 14:15:44,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for d808f9db952c342c04ce9909916ad2b0 in 146ms, sequenceid=13, compaction requested=false; wal=null 2023-07-12 14:15:44,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-12 14:15:44,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/0000000000000000013 2023-07-12 14:15:44,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:15:44,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-12 14:15:44,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened d808f9db952c342c04ce9909916ad2b0; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59306148, jitterRate=-0.11626952886581421}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 14:15:44,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for d808f9db952c342c04ce9909916ad2b0: 2023-07-12 14:15:44,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689171342552.d808f9db952c342c04ce9909916ad2b0., pid=16, masterSystemTime=1689171344350 2023-07-12 14:15:44,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:44,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(158): Opened t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:44,547 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=d808f9db952c342c04ce9909916ad2b0, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:44,548 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689171344547"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689171344547"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689171344547"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689171344547"}]},"ts":"1689171344547"} 2023-07-12 14:15:44,554 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-12 14:15:44,554 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure d808f9db952c342c04ce9909916ad2b0, server=jenkins-hbase17.apache.org,34175,1689171338924 in 354 msec 2023-07-12 14:15:44,557 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-12 14:15:44,557 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase17.apache.org,46441,1689171338873 after splitting done 2023-07-12 14:15:44,557 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN in 516 msec 2023-07-12 14:15:44,557 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase17.apache.org,46441,1689171338873 from processing; numProcessing=0 2023-07-12 14:15:44,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase17.apache.org,46441,1689171338873, splitWal=true, meta=false in 872 msec 2023-07-12 14:15:44,637 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f6eb20c to 127.0.0.1:51038 2023-07-12 14:15:44,637 DEBUG [Listener at localhost.localdomain/36341] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:15:44,638 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-12 14:15:44,655 INFO [Listener at localhost.localdomain/36341] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=268 (was 293), OpenFileDescriptor=564 (was 580), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=473 (was 473), ProcessCount=175 (was 175), AvailableMemoryMB=2529 (was 2600) 2023-07-12 14:15:44,670 INFO [Listener at localhost.localdomain/36341] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=268, OpenFileDescriptor=564, MaxFileDescriptor=60000, SystemLoadAverage=473, ProcessCount=175, AvailableMemoryMB=2528 2023-07-12 14:15:44,671 INFO [Listener at localhost.localdomain/36341] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase17:46441 2023-07-12 14:15:44,684 INFO [Listener at localhost.localdomain/36341] client.ConnectionUtils(127): regionserver/jenkins-hbase17:0 server-side Connection retries=45 2023-07-12 14:15:44,684 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:44,684 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:44,684 INFO [Listener at localhost.localdomain/36341] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 14:15:44,685 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 14:15:44,685 INFO [Listener at localhost.localdomain/36341] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 14:15:44,685 INFO [Listener at localhost.localdomain/36341] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 14:15:44,688 INFO [Listener at localhost.localdomain/36341] ipc.NettyRpcServer(120): Bind to /136.243.18.41:40923 2023-07-12 14:15:44,688 INFO [Listener at localhost.localdomain/36341] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 14:15:44,689 DEBUG [Listener at localhost.localdomain/36341] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 14:15:44,690 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:44,692 INFO [Listener at localhost.localdomain/36341] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 14:15:44,693 INFO [Listener at localhost.localdomain/36341] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40923 connecting to ZooKeeper ensemble=127.0.0.1:51038 2023-07-12 14:15:44,696 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:409230x0, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 14:15:44,697 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(162): regionserver:409230x0, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 14:15:44,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:40923-0x10159e299f20008 connected 2023-07-12 14:15:44,699 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(162): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 14:15:44,700 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ZKUtil(164): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 14:15:44,702 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40923 2023-07-12 14:15:44,704 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40923 2023-07-12 14:15:44,704 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40923 2023-07-12 14:15:44,705 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40923 2023-07-12 14:15:44,705 DEBUG [Listener at localhost.localdomain/36341] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40923 2023-07-12 14:15:44,707 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(951): ClusterId : 70f49905-7e5f-4cbe-8336-b641ab793488 2023-07-12 14:15:44,708 DEBUG [RS:2;jenkins-hbase17:40923] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 14:15:44,710 DEBUG [RS:2;jenkins-hbase17:40923] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 14:15:44,710 DEBUG [RS:2;jenkins-hbase17:40923] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 14:15:44,712 DEBUG [RS:2;jenkins-hbase17:40923] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 14:15:44,714 DEBUG [RS:2;jenkins-hbase17:40923] zookeeper.ReadOnlyZKClient(139): Connect 0x5b59dd26 to 127.0.0.1:51038 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 14:15:44,724 DEBUG [RS:2;jenkins-hbase17:40923] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@559e283d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 14:15:44,725 DEBUG [RS:2;jenkins-hbase17:40923] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51a1b656, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase17.apache.org/136.243.18.41:0 2023-07-12 14:15:44,733 DEBUG [RS:2;jenkins-hbase17:40923] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase17:40923 2023-07-12 14:15:44,734 INFO [RS:2;jenkins-hbase17:40923] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 14:15:44,734 INFO [RS:2;jenkins-hbase17:40923] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 14:15:44,734 DEBUG [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 14:15:44,735 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase17.apache.org,44183,1689171337725 with isa=jenkins-hbase17.apache.org/136.243.18.41:40923, startcode=1689171344683 2023-07-12 14:15:44,735 DEBUG [RS:2;jenkins-hbase17:40923] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 14:15:44,739 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:60821, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 14:15:44,740 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44183] master.ServerManager(394): Registering regionserver=jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:15:44,741 DEBUG [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8 2023-07-12 14:15:44,741 DEBUG [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:39107 2023-07-12 14:15:44,741 DEBUG [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 14:15:44,743 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:15:44,743 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:15:44,743 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase17.apache.org,40923,1689171344683] 2023-07-12 14:15:44,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:44,757 DEBUG [RS:2;jenkins-hbase17:40923] zookeeper.ZKUtil(162): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:15:44,757 WARN [RS:2;jenkins-hbase17:40923] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 14:15:44,757 INFO [RS:2;jenkins-hbase17:40923] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 14:15:44,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:15:44,757 DEBUG [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:15:44,764 DEBUG [RS:2;jenkins-hbase17:40923] zookeeper.ZKUtil(162): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:15:44,769 DEBUG [RS:2;jenkins-hbase17:40923] zookeeper.ZKUtil(162): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:15:44,770 DEBUG [RS:2;jenkins-hbase17:40923] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 14:15:44,771 INFO [RS:2;jenkins-hbase17:40923] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 14:15:44,784 INFO [RS:2;jenkins-hbase17:40923] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 14:15:44,785 INFO [RS:2;jenkins-hbase17:40923] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 14:15:44,786 INFO [RS:2;jenkins-hbase17:40923] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:44,786 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 14:15:44,788 INFO [RS:2;jenkins-hbase17:40923] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:44,788 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:44,789 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:44,789 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:44,789 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:44,789 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:44,789 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0, corePoolSize=2, maxPoolSize=2 2023-07-12 14:15:44,789 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:44,789 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:44,789 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:44,790 DEBUG [RS:2;jenkins-hbase17:40923] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-07-12 14:15:44,800 INFO [RS:2;jenkins-hbase17:40923] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:44,800 INFO [RS:2;jenkins-hbase17:40923] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:44,800 INFO [RS:2;jenkins-hbase17:40923] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:44,812 INFO [RS:2;jenkins-hbase17:40923] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 14:15:44,812 INFO [RS:2;jenkins-hbase17:40923] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,40923,1689171344683-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 14:15:44,831 INFO [RS:2;jenkins-hbase17:40923] regionserver.Replication(203): jenkins-hbase17.apache.org,40923,1689171344683 started 2023-07-12 14:15:44,831 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1637): Serving as jenkins-hbase17.apache.org,40923,1689171344683, RpcServer on jenkins-hbase17.apache.org/136.243.18.41:40923, sessionid=0x10159e299f20008 2023-07-12 14:15:44,831 INFO [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase17:40923,5,FailOnTimeoutGroup] 2023-07-12 14:15:44,831 DEBUG [RS:2;jenkins-hbase17:40923] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 14:15:44,831 INFO [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:15:44,832 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(177): START ************ test3686b 2023-07-12 14:15:44,832 DEBUG [RS:2;jenkins-hbase17:40923] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:15:44,832 DEBUG [RS:2;jenkins-hbase17:40923] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,40923,1689171344683' 2023-07-12 14:15:44,832 DEBUG [RS:2;jenkins-hbase17:40923] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 14:15:44,833 DEBUG [RS:2;jenkins-hbase17:40923] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 14:15:44,833 DEBUG [RS:2;jenkins-hbase17:40923] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 14:15:44,833 DEBUG [RS:2;jenkins-hbase17:40923] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 14:15:44,833 DEBUG [RS:2;jenkins-hbase17:40923] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:15:44,833 DEBUG [RS:2;jenkins-hbase17:40923] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,40923,1689171344683' 2023-07-12 14:15:44,833 DEBUG [RS:2;jenkins-hbase17:40923] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 14:15:44,834 DEBUG [RS:2;jenkins-hbase17:40923] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 14:15:44,834 DEBUG [RS:2;jenkins-hbase17:40923] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 14:15:44,834 INFO [RS:2;jenkins-hbase17:40923] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 14:15:44,834 INFO [RS:2;jenkins-hbase17:40923] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 14:15:44,835 DEBUG [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-12 14:15:44,835 DEBUG [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:15:44,836 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ReadOnlyZKClient(139): Connect 0x5905de0c to 127.0.0.1:51038 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 14:15:44,843 DEBUG [Listener at localhost.localdomain/36341] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@196d90fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 14:15:44,847 DEBUG [hconnection-0x77f3d5fa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 14:15:44,853 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:56274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 14:15:44,937 INFO [RS:2;jenkins-hbase17:40923] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C40923%2C1689171344683, suffix=, logDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,40923,1689171344683, archiveDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs, maxLogs=32 2023-07-12 14:15:44,971 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK] 2023-07-12 14:15:44,972 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK] 2023-07-12 14:15:44,997 INFO [RS:2;jenkins-hbase17:40923] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,40923,1689171344683/jenkins-hbase17.apache.org%2C40923%2C1689171344683.1689171344939 2023-07-12 14:15:45,008 DEBUG [RS:2;jenkins-hbase17:40923] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK], DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK]] 2023-07-12 14:15:46,751 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 14:15:46,866 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 14:15:46,869 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 14:15:47,889 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 14:15:47,890 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 14:15:59,860 INFO [regionserver/jenkins-hbase17:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 5227649225629630483 expired clientIPAndPort=136.243.18.41:56274, userName=jenkins, regionInfo=t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:01,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34175] regionserver.RSRpcServices(3094): Client tried to access missing scanner 5227649225629630483 2023-07-12 14:16:01,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34175] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 136.243.18.41:56274 deadline: 1689171421861, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '5227649225629630483'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-12 14:16:01,864 DEBUG [Listener at localhost.localdomain/36341] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '5227649225629630483'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '5227649225629630483'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '5227649225629630483'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-12 14:16:01,872 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5905de0c to 127.0.0.1:51038 2023-07-12 14:16:01,872 DEBUG [Listener at localhost.localdomain/36341] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:16:01,872 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-12 14:16:01,908 INFO [Listener at localhost.localdomain/36341] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=295 (was 268) Potentially hanging thread: RS:2;jenkins-hbase17:40923-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51038@0x5b59dd26-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS:2;jenkins-hbase17:40923 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost.localdomain:39107 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51038@0x5b59dd26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1943232432.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51038@0x5905de0c-SendThread(127.0.0.1:51038) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: PacketResponder: BP-49469149-136.243.18.41-1689171334293:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51038@0x5905de0c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.IOUtil.drain(Native Method) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51038@0x5905de0c java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1943232432.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x77f3d5fa-shared-pool-1 sun.misc.Unsafe.unpark(Native Method) java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141) java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662) java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264) java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457) java.util.concurrent.ThreadPoolExecutor.tryTerminate(ThreadPoolExecutor.java:721) java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1014) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase17:40923Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8-prefix:jenkins-hbase17.apache.org,40923,1689171344683 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-49469149-136.243.18.41-1689171334293:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-573613859_17 at /127.0.0.1:37804 [Receiving block BP-49469149-136.243.18.41-1689171334293:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-573613859_17 at /127.0.0.1:57720 [Receiving block BP-49469149-136.243.18.41-1689171334293:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40923 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51038@0x5b59dd26-SendThread(127.0.0.1:51038) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) - Thread LEAK? -, OpenFileDescriptor=571 (was 564) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=385 (was 473), ProcessCount=175 (was 175), AvailableMemoryMB=3006 (was 2528) - AvailableMemoryMB LEAK? - 2023-07-12 14:16:01,937 INFO [Listener at localhost.localdomain/36341] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=294, OpenFileDescriptor=571, MaxFileDescriptor=60000, SystemLoadAverage=385, ProcessCount=175, AvailableMemoryMB=3004 2023-07-12 14:16:01,938 INFO [Listener at localhost.localdomain/36341] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase17:46441 2023-07-12 14:16:01,938 INFO [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase17.apache.org,46441,1689171338873 2023-07-12 14:16:01,938 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(104): START************ test2772 2023-07-12 14:16:01,943 DEBUG [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-12 14:16:01,943 DEBUG [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:01,944 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ReadOnlyZKClient(139): Connect 0x1484c7ba to 127.0.0.1:51038 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 14:16:01,953 DEBUG [Listener at localhost.localdomain/36341] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b7439b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 14:16:01,954 ERROR [Listener at localhost.localdomain/36341] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase17.apache.org,34175,1689171338924: die! ***** 2023-07-12 14:16:01,954 ERROR [Listener at localhost.localdomain/36341] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 14:16:01,954 DEBUG [Listener at localhost.localdomain/36341] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 14:16:01,955 DEBUG [Listener at localhost.localdomain/36341] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 14:16:01,972 DEBUG [Listener at localhost.localdomain/36341] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 14:16:01,974 DEBUG [Listener at localhost.localdomain/36341] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 14:16:02,055 INFO [Listener at localhost.localdomain/36341] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1005584384, "init": 524288000, "max": 2051014656, "used": 264238368 }, "NonHeapMemoryUsage": { "committed": 122249216, "init": 2555904, "max": -1, "used": 119785040 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase17.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11984, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 196, "ResponseSize_max": 680, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 197, "ResponseSize_75th_percentile": 677, "ResponseSize_90th_percentile": 679, "ResponseSize_95th_percentile": 679, "ResponseSize_98th_percentile": 679, "ResponseSize_99th_percentile": 679, "ResponseSize_99.9th_percentile": 679, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16696 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase17.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:51038", "tag.serverName": "jenkins-hbase17.apache.org,34175,1689171338924", "tag.clusterId": "70f49905-7e5f-4cbe-8336-b641ab793488", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase17.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 1, "maxStoreFileCount": 1, "memStoreSize": 3312, "storeFileSize": 4926, "maxStoreFileAge": 16123, "minStoreFileAge": 16123, "avgStoreFileAge": 16123, "numReferenceFiles": 0, "regionServerStartTime": 1689171338924, "averageRegionSize": 2746, "storeFileIndexSize": 320, "staticIndexSize": 31, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 1863680, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 46, "totalRowActionRequestCount": 57, "readRequestCount": 43, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 10, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 22, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 10, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 270, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 21, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 10, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 21, "ScanSize_min": 155, "ScanSize_max": 655, "ScanSize_mean": 405, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 654, "ScanSize_90th_percentile": 654, "ScanSize_95th_percentile": 654, "ScanSize_98th_percentile": 654, "ScanSize_99th_percentile": 654, "ScanSize_99.9th_percentile": 654, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 270, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.52584090349618, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.1225961032951477, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-12 14:16:02,056 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44183] master.MasterRpcServices(612): jenkins-hbase17.apache.org,34175,1689171338924 reported a fatal error: ***** ABORTING region server jenkins-hbase17.apache.org,34175,1689171338924: die! ***** 2023-07-12 14:16:02,064 INFO [Listener at localhost.localdomain/36341] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase17.apache.org,34175,1689171338924' ***** 2023-07-12 14:16:02,066 INFO [Listener at localhost.localdomain/36341] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-12 14:16:02,066 INFO [RS:1;jenkins-hbase17:34175] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 14:16:02,066 INFO [RS:1;jenkins-hbase17:34175] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 14:16:02,066 INFO [RS:1;jenkins-hbase17:34175] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 14:16:02,066 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(3305): Received CLOSE for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:02,066 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 14:16:02,067 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(3305): Received CLOSE for c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:16:02,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing d808f9db952c342c04ce9909916ad2b0, disabling compactions & flushes 2023-07-12 14:16:02,068 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1141): aborting server jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:16:02,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:02,068 DEBUG [RS:1;jenkins-hbase17:34175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51d36f56 to 127.0.0.1:51038 2023-07-12 14:16:02,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:02,068 DEBUG [RS:1;jenkins-hbase17:34175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:16:02,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. after waiting 0 ms 2023-07-12 14:16:02,068 INFO [RS:1;jenkins-hbase17:34175] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 14:16:02,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:02,069 INFO [RS:1;jenkins-hbase17:34175] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 14:16:02,069 INFO [RS:1;jenkins-hbase17:34175] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 14:16:02,069 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 14:16:02,073 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 14:16:02,074 DEBUG [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1478): Online Regions={d808f9db952c342c04ce9909916ad2b0=t,,1689171342552.d808f9db952c342c04ce9909916ad2b0., 1588230740=hbase:meta,,1.1588230740, c0e52329bdbaee97aeea58f1b40da9f0=hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.} 2023-07-12 14:16:02,074 DEBUG [hconnection-0x25f9acec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 14:16:02,075 DEBUG [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1504): Waiting on 1588230740, c0e52329bdbaee97aeea58f1b40da9f0, d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:02,074 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 14:16:02,075 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 14:16:02,075 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 14:16:02,075 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 14:16:02,075 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 14:16:02,080 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1825): Memstore data size is 3234 in region hbase:meta,,1.1588230740 2023-07-12 14:16:02,080 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 14:16:02,082 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 14:16:02,082 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 14:16:02,082 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 14:16:02,085 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:51122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 14:16:02,085 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34175] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,34175,1689171338924 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 14:16:02,086 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34175] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 136.243.18.41:51122 deadline: 1689171422085, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,34175,1689171338924 aborting 2023-07-12 14:16:02,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:02,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for d808f9db952c342c04ce9909916ad2b0: 2023-07-12 14:16:02,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:02,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing c0e52329bdbaee97aeea58f1b40da9f0, disabling compactions & flushes 2023-07-12 14:16:02,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:02,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:02,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. after waiting 0 ms 2023-07-12 14:16:02,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:02,112 INFO [regionserver/jenkins-hbase17:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 14:16:02,128 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:02,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:02,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for c0e52329bdbaee97aeea58f1b40da9f0: 2023-07-12 14:16:02,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:02,192 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34175] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,34175,1689171338924 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 14:16:02,192 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34175] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 136.243.18.41:51122 deadline: 1689171422192, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,34175,1689171338924 aborting 2023-07-12 14:16:02,275 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1170): stopping server jenkins-hbase17.apache.org,34175,1689171338924; all regions closed. 2023-07-12 14:16:02,303 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900 not finished, retry = 0 2023-07-12 14:16:02,396 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34175] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,34175,1689171338924 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 14:16:02,396 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34175] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 111 connection: 136.243.18.41:51122 deadline: 1689171422396, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase17.apache.org,34175,1689171338924 aborting 2023-07-12 14:16:02,405 DEBUG [RS:1;jenkins-hbase17:34175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:16:02,406 INFO [RS:1;jenkins-hbase17:34175] regionserver.LeaseManager(133): Closed leases 2023-07-12 14:16:02,406 INFO [RS:1;jenkins-hbase17:34175] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase17:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 14:16:02,407 INFO [regionserver/jenkins-hbase17:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 14:16:02,408 INFO [RS:1;jenkins-hbase17:34175] ipc.NettyRpcServer(158): Stopping server on /136.243.18.41:34175 2023-07-12 14:16:02,414 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:16:02,415 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:16:02,415 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:16:02,416 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 2023-07-12 14:16:02,417 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:16:02,422 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase17.apache.org,34175,1689171338924] 2023-07-12 14:16:02,422 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase17.apache.org,34175,1689171338924; numProcessing=1 2023-07-12 14:16:02,522 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 14:16:02,522 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:34175-0x10159e299f20002, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 14:16:02,523 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase17.apache.org,34175,1689171338924 already deleted, retry=false 2023-07-12 14:16:02,523 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase17.apache.org,34175,1689171338924 on jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:16:02,523 INFO [RS:1;jenkins-hbase17:34175] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase17.apache.org,34175,1689171338924; zookeeper connection closed. 2023-07-12 14:16:02,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:02,525 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase17.apache.org,34175,1689171338924, splitWal=true, meta=true 2023-07-12 14:16:02,525 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase17.apache.org,34175,1689171338924 (carryingMeta=true) jenkins-hbase17.apache.org,34175,1689171338924/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@44a26124[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-12 14:16:02,533 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase17.apache.org,34175,1689171338924 znode expired, triggering replicatorRemoved event 2023-07-12 14:16:02,534 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@e856a36] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@e856a36 2023-07-12 14:16:02,535 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase17.apache.org,34175,1689171338924, splitWal=true, meta=true 2023-07-12 14:16:02,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:02,538 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase17.apache.org,34175,1689171338924, splitWal=true, meta=true, isMeta: true 2023-07-12 14:16:02,583 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting 2023-07-12 14:16:02,586 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase17.apache.org,34175,1689171338924 WAL count=1, meta=true 2023-07-12 14:16:02,586 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta}] 2023-07-12 14:16:02,594 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:02,595 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta, worker=jenkins-hbase17.apache.org,40923,1689171344683}] 2023-07-12 14:16:02,701 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase17.apache.org/136.243.18.41:34175 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase17.apache.org/136.243.18.41:34175 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 14:16:02,703 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase17.apache.org/136.243.18.41:34175 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase17.apache.org/136.243.18.41:34175 2023-07-12 14:16:02,751 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:02,751 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 14:16:02,753 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:54712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 14:16:02,754 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40923] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-12 14:16:02,788 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta, size=4.0 K (4130bytes) 2023-07-12 14:16:02,788 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta 2023-07-12 14:16:02,789 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta after 1ms 2023-07-12 14:16:02,800 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta took 12ms 2023-07-12 14:16:02,813 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-12 14:16:02,815 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta so closing down 2023-07-12 14:16:02,815 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 14:16:02,820 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta.temp 2023-07-12 14:16:02,821 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 14:16:02,840 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-12 14:16:02,844 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta.temp to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-12 14:16:02,844 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 36 ms; skipped=1; WAL=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta, size=4.0 K, length=4130, corrupted=false, cancelled=false 2023-07-12 14:16:02,845 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta, journal: Splitting hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta, size=4.0 K (4130bytes) at 1689171362788Finishing writing output for hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta so closing down at 1689171362815 (+27 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta.temp at 1689171362820 (+5 ms)3 split writer threads finished at 1689171362821 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689171362840 (+19 ms)Rename recovered edits hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta.temp to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689171362844 (+4 ms)Processed 13 edits across 1 Regions in 36 ms; skipped=1; WAL=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta, size=4.0 K, length=4130, corrupted=false, cancelled=false at 1689171362844 2023-07-12 14:16:02,845 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta 2023-07-12 14:16:02,845 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-12 14:16:02,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] master.HMaster(3641): Remote procedure done, pid=19 2023-07-12 14:16:02,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs 2023-07-12 14:16:02,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-12 14:16:02,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta, worker=jenkins-hbase17.apache.org,40923,1689171344683 in 266 msec 2023-07-12 14:16:02,868 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:02,877 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-12 14:16:02,877 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase17.apache.org%2C34175%2C1689171338924.meta.1689171341135.meta, worker=jenkins-hbase17.apache.org,40923,1689171344683 in 283 msec 2023-07-12 14:16:02,888 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting dir is empty, no logs to split. 2023-07-12 14:16:02,888 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase17.apache.org,34175,1689171338924 WAL count=0, meta=true 2023-07-12 14:16:02,888 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase17.apache.org,34175,1689171338924 WAL splitting is done? wals=0, meta=true 2023-07-12 14:16:02,891 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 14:16:02,893 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 14:16:02,895 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 14:16:03,045 DEBUG [jenkins-hbase17:44183] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 14:16:03,046 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase17.apache.org,40923,1689171344683, state=OPENING 2023-07-12 14:16:03,047 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 14:16:03,047 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase17.apache.org,40923,1689171344683}] 2023-07-12 14:16:03,047 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 14:16:03,206 WARN [ReadOnlyZKClient-127.0.0.1:51038@0x1484c7ba] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-12 14:16:03,207 DEBUG [hconnection-0x25f9acec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 14:16:03,221 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:54720, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 14:16:03,221 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40923] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Scan size: 111 connection: 136.243.18.41:54720 deadline: 1689171423221, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:03,235 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 14:16:03,235 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 14:16:03,239 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C40923%2C1689171344683.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,40923,1689171344683, archiveDir=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs, maxLogs=32 2023-07-12 14:16:03,272 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK] 2023-07-12 14:16:03,273 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK] 2023-07-12 14:16:03,289 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,40923,1689171344683/jenkins-hbase17.apache.org%2C40923%2C1689171344683.meta.1689171363240.meta 2023-07-12 14:16:03,289 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44245,DS-ab4ecc90-009e-42aa-b697-0f0860452658,DISK], DatanodeInfoWithStorage[127.0.0.1:42549,DS-3c98cb4e-0a14-440b-97e9-281b345f6a36,DISK]] 2023-07-12 14:16:03,289 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:16:03,290 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 14:16:03,290 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 14:16:03,291 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 14:16:03,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 14:16:03,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:16:03,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 14:16:03,292 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 14:16:03,298 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 14:16:03,304 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/info 2023-07-12 14:16:03,305 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/info 2023-07-12 14:16:03,305 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 14:16:03,306 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:16:03,307 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 14:16:03,312 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/rep_barrier 2023-07-12 14:16:03,312 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/rep_barrier 2023-07-12 14:16:03,313 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 14:16:03,314 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:16:03,314 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 14:16:03,315 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/table 2023-07-12 14:16:03,315 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/table 2023-07-12 14:16:03,316 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 14:16:03,318 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:16:03,320 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740 2023-07-12 14:16:03,324 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740 2023-07-12 14:16:03,325 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-12 14:16:03,337 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-12 14:16:03,338 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.16 KB heapSize=6.27 KB 2023-07-12 14:16:03,393 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/.tmp/info/496c7cba7e61476191dc3a0d05f240dc 2023-07-12 14:16:03,426 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/.tmp/table/65bac609d247459a99ab7160d851b860 2023-07-12 14:16:03,441 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/.tmp/info/496c7cba7e61476191dc3a0d05f240dc as hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/info/496c7cba7e61476191dc3a0d05f240dc 2023-07-12 14:16:03,455 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/info/496c7cba7e61476191dc3a0d05f240dc, entries=28, sequenceid=15, filesize=7.7 K 2023-07-12 14:16:03,457 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/.tmp/table/65bac609d247459a99ab7160d851b860 as hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/table/65bac609d247459a99ab7160d851b860 2023-07-12 14:16:03,470 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/table/65bac609d247459a99ab7160d851b860, entries=4, sequenceid=15, filesize=4.7 K 2023-07-12 14:16:03,470 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.16 KB/3234, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 132ms, sequenceid=15, compaction requested=false; wal=null 2023-07-12 14:16:03,472 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-12 14:16:03,476 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 14:16:03,480 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 14:16:03,484 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-12 14:16:03,485 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68654733, jitterRate=0.023035243153572083}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 14:16:03,485 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 14:16:03,487 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689171363204 2023-07-12 14:16:03,489 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 14:16:03,490 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 14:16:03,491 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase17.apache.org,40923,1689171344683, state=OPEN 2023-07-12 14:16:03,492 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 14:16:03,492 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 14:16:03,496 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-12 14:16:03,496 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase17.apache.org,40923,1689171344683 in 445 msec 2023-07-12 14:16:03,501 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-12 14:16:03,501 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase17.apache.org,34175,1689171338924 had 3 regions 2023-07-12 14:16:03,501 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 605 msec 2023-07-12 14:16:03,503 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase17.apache.org,34175,1689171338924, splitWal=true, meta=true, isMeta: false 2023-07-12 14:16:03,506 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase17.apache.org,34175,1689171338924 WAL count=1, meta=false 2023-07-12 14:16:03,506 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900}] 2023-07-12 14:16:03,509 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:03,515 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900, worker=jenkins-hbase17.apache.org,40923,1689171344683}] 2023-07-12 14:16:03,669 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40923] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-12 14:16:03,687 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900, size=901 (901bytes) 2023-07-12 14:16:03,687 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900 2023-07-12 14:16:03,688 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900 after 1ms 2023-07-12 14:16:03,693 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] wal.WALSplitter(311): Open hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900 took 7ms 2023-07-12 14:16:03,697 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] wal.WALSplitter(346): Last flushed sequenceid for c0e52329bdbaee97aeea58f1b40da9f0: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-12 14:16:03,700 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] wal.WALSplitter(346): Last flushed sequenceid for d808f9db952c342c04ce9909916ad2b0: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-12 14:16:03,701 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900 so closing down 2023-07-12 14:16:03,701 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 14:16:03,705 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900.temp 2023-07-12 14:16:03,705 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 14:16:03,729 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-12 14:16:03,731 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900.temp to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000005 2023-07-12 14:16:03,731 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 38 ms; skipped=2; WAL=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900, size=901, length=901, corrupted=false, cancelled=false 2023-07-12 14:16:03,731 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900, journal: Splitting hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900, size=901 (901bytes) at 1689171363687Finishing writing output for hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900 so closing down at 1689171363701 (+14 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900.temp at 1689171363705 (+4 ms)3 split writer threads finished at 1689171363705Closed recovered edits writer path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689171363729 (+24 ms)Rename recovered edits hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000004-jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900.temp to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000005 at 1689171363731 (+2 ms)Processed 4 edits across 1 Regions in 38 ms; skipped=2; WAL=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900, size=901, length=901, corrupted=false, cancelled=false at 1689171363731 2023-07-12 14:16:03,731 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900 2023-07-12 14:16:03,731 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-12 14:16:03,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] master.HMaster(3641): Remote procedure done, pid=23 2023-07-12 14:16:03,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44183] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting/jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900 to hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs 2023-07-12 14:16:03,746 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 14:16:03,746 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900, worker=jenkins-hbase17.apache.org,40923,1689171344683 in 224 msec 2023-07-12 14:16:03,748 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:03,751 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-12 14:16:03,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase17.apache.org%2C34175%2C1689171338924.1689171340900, worker=jenkins-hbase17.apache.org,40923,1689171344683 in 243 msec 2023-07-12 14:16:03,754 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/WALs/jenkins-hbase17.apache.org,34175,1689171338924-splitting dir is empty, no logs to split. 2023-07-12 14:16:03,754 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase17.apache.org,34175,1689171338924 WAL count=0, meta=false 2023-07-12 14:16:03,754 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase17.apache.org,34175,1689171338924 WAL splitting is done? wals=0, meta=false 2023-07-12 14:16:03,757 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0e52329bdbaee97aeea58f1b40da9f0, ASSIGN}] 2023-07-12 14:16:03,759 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN 2023-07-12 14:16:03,759 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c0e52329bdbaee97aeea58f1b40da9f0, ASSIGN 2023-07-12 14:16:03,760 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c0e52329bdbaee97aeea58f1b40da9f0, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 14:16:03,760 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 14:16:03,910 DEBUG [jenkins-hbase17:44183] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 14:16:03,911 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c0e52329bdbaee97aeea58f1b40da9f0, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:03,911 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=d808f9db952c342c04ce9909916ad2b0, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:03,911 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689171363911"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689171363911"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689171363911"}]},"ts":"1689171363911"} 2023-07-12 14:16:03,911 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689171363911"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689171363911"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689171363911"}]},"ts":"1689171363911"} 2023-07-12 14:16:03,912 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase17.apache.org/136.243.18.41:34175 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase17.apache.org/136.243.18.41:34175 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 14:16:03,914 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase17.apache.org/136.243.18.41:34175 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase17.apache.org/136.243.18.41:34175 2023-07-12 14:16:04,016 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 14:16:04,017 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:54726, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 14:16:04,020 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=24, state=RUNNABLE; OpenRegionProcedure d808f9db952c342c04ce9909916ad2b0, server=jenkins-hbase17.apache.org,40923,1689171344683}] 2023-07-12 14:16:04,021 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=25, state=RUNNABLE; OpenRegionProcedure c0e52329bdbaee97aeea58f1b40da9f0, server=jenkins-hbase17.apache.org,40923,1689171344683}] 2023-07-12 14:16:04,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(130): Open t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:04,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d808f9db952c342c04ce9909916ad2b0, NAME => 't,,1689171342552.d808f9db952c342c04ce9909916ad2b0.', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:16:04,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:04,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:16:04,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:04,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:04,180 INFO [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:04,184 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:04,184 DEBUG [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/f 2023-07-12 14:16:04,185 DEBUG [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/f 2023-07-12 14:16:04,185 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c0e52329bdbaee97aeea58f1b40da9f0, NAME => 'hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.', STARTKEY => '', ENDKEY => ''} 2023-07-12 14:16:04,185 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:16:04,185 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 14:16:04,185 INFO [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d808f9db952c342c04ce9909916ad2b0 columnFamilyName f 2023-07-12 14:16:04,185 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:16:04,186 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:16:04,189 INFO [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:16:04,192 DEBUG [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/info 2023-07-12 14:16:04,192 DEBUG [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/info 2023-07-12 14:16:04,193 INFO [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c0e52329bdbaee97aeea58f1b40da9f0 columnFamilyName info 2023-07-12 14:16:04,194 INFO [StoreOpener-c0e52329bdbaee97aeea58f1b40da9f0-1] regionserver.HStore(310): Store=c0e52329bdbaee97aeea58f1b40da9f0/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:16:04,195 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:16:04,201 DEBUG [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/f/1b2b27f8f72d4aa582592210013cd653 2023-07-12 14:16:04,202 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:16:04,202 INFO [StoreOpener-d808f9db952c342c04ce9909916ad2b0-1] regionserver.HStore(310): Store=d808f9db952c342c04ce9909916ad2b0/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 14:16:04,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:04,204 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000005 2023-07-12 14:16:04,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:04,209 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000005 2023-07-12 14:16:04,209 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing c0e52329bdbaee97aeea58f1b40da9f0 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 14:16:04,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:04,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-12 14:16:04,224 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened d808f9db952c342c04ce9909916ad2b0; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67740759, jitterRate=0.009415969252586365}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 14:16:04,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for d808f9db952c342c04ce9909916ad2b0: 2023-07-12 14:16:04,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689171342552.d808f9db952c342c04ce9909916ad2b0., pid=26, masterSystemTime=1689171364173 2023-07-12 14:16:04,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:04,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(158): Opened t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:04,233 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=d808f9db952c342c04ce9909916ad2b0, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:04,233 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689171342552.d808f9db952c342c04ce9909916ad2b0.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689171364233"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689171364233"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689171364233"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689171364233"}]},"ts":"1689171364233"} 2023-07-12 14:16:04,234 DEBUG [hconnection-0x25f9acec-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase17.apache.org/136.243.18.41:34175 this server is in the failed servers list 2023-07-12 14:16:04,247 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=24 2023-07-12 14:16:04,247 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=24, state=SUCCESS; OpenRegionProcedure d808f9db952c342c04ce9909916ad2b0, server=jenkins-hbase17.apache.org,40923,1689171344683 in 216 msec 2023-07-12 14:16:04,251 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=d808f9db952c342c04ce9909916ad2b0, ASSIGN in 490 msec 2023-07-12 14:16:04,268 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/.tmp/info/8f5e76583b5a4af2ba286e6592ecf28a 2023-07-12 14:16:04,278 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/.tmp/info/8f5e76583b5a4af2ba286e6592ecf28a as hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/info/8f5e76583b5a4af2ba286e6592ecf28a 2023-07-12 14:16:04,287 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/info/8f5e76583b5a4af2ba286e6592ecf28a, entries=2, sequenceid=5, filesize=4.8 K 2023-07-12 14:16:04,287 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c0e52329bdbaee97aeea58f1b40da9f0 in 78ms, sequenceid=5, compaction requested=false; wal=null 2023-07-12 14:16:04,289 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/0000000000000000005 2023-07-12 14:16:04,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:16:04,296 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-12 14:16:04,298 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened c0e52329bdbaee97aeea58f1b40da9f0; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72938814, jitterRate=0.08687302470207214}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 14:16:04,298 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for c0e52329bdbaee97aeea58f1b40da9f0: 2023-07-12 14:16:04,299 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0., pid=27, masterSystemTime=1689171364173 2023-07-12 14:16:04,301 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:04,301 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:04,302 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c0e52329bdbaee97aeea58f1b40da9f0, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:04,303 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689171364302"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689171364302"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689171364302"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689171364302"}]},"ts":"1689171364302"} 2023-07-12 14:16:04,308 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=25 2023-07-12 14:16:04,308 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=25, state=SUCCESS; OpenRegionProcedure c0e52329bdbaee97aeea58f1b40da9f0, server=jenkins-hbase17.apache.org,40923,1689171344683 in 284 msec 2023-07-12 14:16:04,312 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=17 2023-07-12 14:16:04,312 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase17.apache.org,34175,1689171338924 after splitting done 2023-07-12 14:16:04,313 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c0e52329bdbaee97aeea58f1b40da9f0, ASSIGN in 551 msec 2023-07-12 14:16:04,313 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase17.apache.org,34175,1689171338924 from processing; numProcessing=0 2023-07-12 14:16:04,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase17.apache.org,34175,1689171338924, splitWal=true, meta=true in 1.7900 sec 2023-07-12 14:16:04,358 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1484c7ba to 127.0.0.1:51038 2023-07-12 14:16:04,358 DEBUG [Listener at localhost.localdomain/36341] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:16:04,358 INFO [Listener at localhost.localdomain/36341] client.TestScannerTimeout(124): END ************ test2772 2023-07-12 14:16:04,376 INFO [Listener at localhost.localdomain/36341] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=276 (was 294), OpenFileDescriptor=583 (was 571) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=385 (was 385), ProcessCount=175 (was 175), AvailableMemoryMB=2846 (was 3004) 2023-07-12 14:16:04,377 INFO [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 14:16:04,377 INFO [Listener at localhost.localdomain/36341] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 14:16:04,378 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43f27a6f to 127.0.0.1:51038 2023-07-12 14:16:04,378 DEBUG [Listener at localhost.localdomain/36341] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:16:04,378 DEBUG [Listener at localhost.localdomain/36341] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 14:16:04,378 DEBUG [Listener at localhost.localdomain/36341] util.JVMClusterUtil(257): Found active master hash=1768762736, stopped=false 2023-07-12 14:16:04,379 INFO [Listener at localhost.localdomain/36341] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:16:04,382 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 14:16:04,382 INFO [Listener at localhost.localdomain/36341] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 14:16:04,382 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 14:16:04,382 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:16:04,383 DEBUG [Listener at localhost.localdomain/36341] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28274dee to 127.0.0.1:51038 2023-07-12 14:16:04,383 DEBUG [Listener at localhost.localdomain/36341] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:16:04,383 INFO [Listener at localhost.localdomain/36341] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase17.apache.org,40923,1689171344683' ***** 2023-07-12 14:16:04,384 INFO [Listener at localhost.localdomain/36341] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 14:16:04,384 INFO [RS:2;jenkins-hbase17:40923] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 14:16:04,384 INFO [RS:2;jenkins-hbase17:40923] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 14:16:04,384 INFO [RS:2;jenkins-hbase17:40923] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 14:16:04,384 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(3305): Received CLOSE for d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:04,384 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 14:16:04,384 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(3305): Received CLOSE for c0e52329bdbaee97aeea58f1b40da9f0 2023-07-12 14:16:04,385 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1144): stopping server jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:04,385 DEBUG [RS:2;jenkins-hbase17:40923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b59dd26 to 127.0.0.1:51038 2023-07-12 14:16:04,385 DEBUG [RS:2;jenkins-hbase17:40923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:16:04,385 INFO [RS:2;jenkins-hbase17:40923] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 14:16:04,385 INFO [RS:2;jenkins-hbase17:40923] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 14:16:04,385 INFO [RS:2;jenkins-hbase17:40923] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 14:16:04,385 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 14:16:04,385 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 14:16:04,385 DEBUG [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1478): Online Regions={d808f9db952c342c04ce9909916ad2b0=t,,1689171342552.d808f9db952c342c04ce9909916ad2b0., 1588230740=hbase:meta,,1.1588230740, c0e52329bdbaee97aeea58f1b40da9f0=hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0.} 2023-07-12 14:16:04,385 DEBUG [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1504): Waiting on 1588230740, c0e52329bdbaee97aeea58f1b40da9f0, d808f9db952c342c04ce9909916ad2b0 2023-07-12 14:16:04,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing d808f9db952c342c04ce9909916ad2b0, disabling compactions & flushes 2023-07-12 14:16:04,386 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 14:16:04,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:04,386 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 14:16:04,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:04,386 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 14:16:04,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. after waiting 0 ms 2023-07-12 14:16:04,386 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 14:16:04,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:04,386 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 14:16:04,387 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-12 14:16:04,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/default/t/d808f9db952c342c04ce9909916ad2b0/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-12 14:16:04,402 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:04,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for d808f9db952c342c04ce9909916ad2b0: 2023-07-12 14:16:04,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed t,,1689171342552.d808f9db952c342c04ce9909916ad2b0. 2023-07-12 14:16:04,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing c0e52329bdbaee97aeea58f1b40da9f0, disabling compactions & flushes 2023-07-12 14:16:04,409 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:04,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:04,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. after waiting 0 ms 2023-07-12 14:16:04,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:04,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/namespace/c0e52329bdbaee97aeea58f1b40da9f0/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-12 14:16:04,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:04,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for c0e52329bdbaee97aeea58f1b40da9f0: 2023-07-12 14:16:04,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689171341543.c0e52329bdbaee97aeea58f1b40da9f0. 2023-07-12 14:16:04,429 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/.tmp/info/d9257b6f1df14c37bb9bc2b0d2be2caa 2023-07-12 14:16:04,439 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/.tmp/info/d9257b6f1df14c37bb9bc2b0d2be2caa as hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/info/d9257b6f1df14c37bb9bc2b0d2be2caa 2023-07-12 14:16:04,448 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/info/d9257b6f1df14c37bb9bc2b0d2be2caa, entries=16, sequenceid=22, filesize=6.4 K 2023-07-12 14:16:04,450 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1796, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 63ms, sequenceid=22, compaction requested=false 2023-07-12 14:16:04,459 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-12 14:16:04,460 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 14:16:04,460 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 14:16:04,460 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 14:16:04,460 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 14:16:04,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 14:16:04,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 14:16:04,586 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1170): stopping server jenkins-hbase17.apache.org,40923,1689171344683; all regions closed. 2023-07-12 14:16:04,595 DEBUG [RS:2;jenkins-hbase17:40923] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs 2023-07-12 14:16:04,596 INFO [RS:2;jenkins-hbase17:40923] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase17.apache.org%2C40923%2C1689171344683.meta:.meta(num 1689171363240) 2023-07-12 14:16:04,605 DEBUG [RS:2;jenkins-hbase17:40923] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/oldWALs 2023-07-12 14:16:04,606 INFO [RS:2;jenkins-hbase17:40923] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase17.apache.org%2C40923%2C1689171344683:(num 1689171344939) 2023-07-12 14:16:04,606 DEBUG [RS:2;jenkins-hbase17:40923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:16:04,606 INFO [RS:2;jenkins-hbase17:40923] regionserver.LeaseManager(133): Closed leases 2023-07-12 14:16:04,606 INFO [RS:2;jenkins-hbase17:40923] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase17:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 14:16:04,606 INFO [regionserver/jenkins-hbase17:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 14:16:04,607 INFO [RS:2;jenkins-hbase17:40923] ipc.NettyRpcServer(158): Stopping server on /136.243.18.41:40923 2023-07-12 14:16:04,609 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:16:04,609 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase17.apache.org,40923,1689171344683 2023-07-12 14:16:04,609 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 14:16:04,610 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase17.apache.org,40923,1689171344683] 2023-07-12 14:16:04,610 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase17.apache.org,40923,1689171344683; numProcessing=1 2023-07-12 14:16:04,611 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase17.apache.org,40923,1689171344683 already deleted, retry=false 2023-07-12 14:16:04,611 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase17.apache.org,40923,1689171344683 expired; onlineServers=0 2023-07-12 14:16:04,611 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase17.apache.org,44183,1689171337725' ***** 2023-07-12 14:16:04,611 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 14:16:04,612 DEBUG [M:0;jenkins-hbase17:44183] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2850274b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase17.apache.org/136.243.18.41:0 2023-07-12 14:16:04,612 INFO [M:0;jenkins-hbase17:44183] regionserver.HRegionServer(1144): stopping server jenkins-hbase17.apache.org,44183,1689171337725 2023-07-12 14:16:04,612 INFO [M:0;jenkins-hbase17:44183] regionserver.HRegionServer(1170): stopping server jenkins-hbase17.apache.org,44183,1689171337725; all regions closed. 2023-07-12 14:16:04,612 DEBUG [M:0;jenkins-hbase17:44183] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 14:16:04,612 DEBUG [M:0;jenkins-hbase17:44183] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 14:16:04,612 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 14:16:04,612 DEBUG [M:0;jenkins-hbase17:44183] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 14:16:04,612 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.large.0-1689171340419] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.large.0-1689171340419,5,FailOnTimeoutGroup] 2023-07-12 14:16:04,612 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.small.0-1689171340426] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.small.0-1689171340426,5,FailOnTimeoutGroup] 2023-07-12 14:16:04,612 INFO [M:0;jenkins-hbase17:44183] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 14:16:04,613 INFO [M:0;jenkins-hbase17:44183] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 14:16:04,613 INFO [M:0;jenkins-hbase17:44183] hbase.ChoreService(369): Chore service for: master/jenkins-hbase17:0 had [] on shutdown 2023-07-12 14:16:04,613 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 14:16:04,613 DEBUG [M:0;jenkins-hbase17:44183] master.HMaster(1512): Stopping service threads 2023-07-12 14:16:04,613 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 14:16:04,613 INFO [M:0;jenkins-hbase17:44183] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 14:16:04,613 ERROR [M:0;jenkins-hbase17:44183] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] 2023-07-12 14:16:04,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master Thread[IPC Client (696426968) connection to localhost.localdomain/127.0.0.1:39107 from jenkins,5,PEWorkerGroup] 2023-07-12 14:16:04,614 INFO [M:0;jenkins-hbase17:44183] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 14:16:04,614 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 14:16:04,614 DEBUG [M:0;jenkins-hbase17:44183] zookeeper.ZKUtil(398): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 14:16:04,614 WARN [M:0;jenkins-hbase17:44183] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 14:16:04,615 INFO [M:0;jenkins-hbase17:44183] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 14:16:04,615 INFO [M:0;jenkins-hbase17:44183] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 14:16:04,615 DEBUG [M:0;jenkins-hbase17:44183] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 14:16:04,615 INFO [M:0;jenkins-hbase17:44183] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 14:16:04,615 DEBUG [M:0;jenkins-hbase17:44183] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 14:16:04,615 DEBUG [M:0;jenkins-hbase17:44183] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 14:16:04,615 DEBUG [M:0;jenkins-hbase17:44183] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 14:16:04,616 INFO [M:0;jenkins-hbase17:44183] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=91.61 KB heapSize=110.32 KB 2023-07-12 14:16:04,631 INFO [M:0;jenkins-hbase17:44183] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=91.61 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/46cd12b83f5e4838b67227175a9eae10 2023-07-12 14:16:04,639 DEBUG [M:0;jenkins-hbase17:44183] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/46cd12b83f5e4838b67227175a9eae10 as hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/46cd12b83f5e4838b67227175a9eae10 2023-07-12 14:16:04,647 INFO [M:0;jenkins-hbase17:44183] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39107/user/jenkins/test-data/bf2c6982-1d63-2da9-1dd8-fc5b35ea00f8/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/46cd12b83f5e4838b67227175a9eae10, entries=27, sequenceid=212, filesize=8.2 K 2023-07-12 14:16:04,648 INFO [M:0;jenkins-hbase17:44183] regionserver.HRegion(2948): Finished flush of dataSize ~91.61 KB/93812, heapSize ~110.30 KB/112952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 33ms, sequenceid=212, compaction requested=false 2023-07-12 14:16:04,650 INFO [M:0;jenkins-hbase17:44183] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 14:16:04,650 DEBUG [M:0;jenkins-hbase17:44183] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 14:16:04,656 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 14:16:04,656 INFO [M:0;jenkins-hbase17:44183] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 14:16:04,657 INFO [M:0;jenkins-hbase17:44183] ipc.NettyRpcServer(158): Stopping server on /136.243.18.41:44183 2023-07-12 14:16:04,659 DEBUG [M:0;jenkins-hbase17:44183] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase17.apache.org,44183,1689171337725 already deleted, retry=false 2023-07-12 14:16:04,761 INFO [M:0;jenkins-hbase17:44183] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase17.apache.org,44183,1689171337725; zookeeper connection closed. 2023-07-12 14:16:04,761 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 14:16:04,762 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): master:44183-0x10159e299f20000, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 14:16:04,781 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 14:16:04,782 INFO [RS:2;jenkins-hbase17:40923] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase17.apache.org,40923,1689171344683; zookeeper connection closed. 2023-07-12 14:16:04,782 DEBUG [Listener at localhost.localdomain/36341-EventThread] zookeeper.ZKWatcher(600): regionserver:40923-0x10159e299f20008, quorum=127.0.0.1:51038, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 14:16:04,784 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@341b3294] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@341b3294 2023-07-12 14:16:04,784 INFO [Listener at localhost.localdomain/36341] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-12 14:16:04,785 WARN [Listener at localhost.localdomain/36341] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 14:16:04,789 INFO [Listener at localhost.localdomain/36341] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 14:16:04,798 WARN [BP-49469149-136.243.18.41-1689171334293 heartbeating to localhost.localdomain/127.0.0.1:39107] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 14:16:04,798 WARN [BP-49469149-136.243.18.41-1689171334293 heartbeating to localhost.localdomain/127.0.0.1:39107] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-49469149-136.243.18.41-1689171334293 (Datanode Uuid 5ee61f3b-3d89-4c08-ab5d-f8a4364a1745) service to localhost.localdomain/127.0.0.1:39107 2023-07-12 14:16:04,800 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79/dfs/data/data3/current/BP-49469149-136.243.18.41-1689171334293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 14:16:04,800 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79/dfs/data/data4/current/BP-49469149-136.243.18.41-1689171334293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 14:16:04,807 INFO [regionserver/jenkins-hbase17:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 14:16:04,809 WARN [Listener at localhost.localdomain/36341] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 14:16:04,813 INFO [Listener at localhost.localdomain/36341] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 14:16:04,919 WARN [BP-49469149-136.243.18.41-1689171334293 heartbeating to localhost.localdomain/127.0.0.1:39107] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 14:16:04,919 WARN [BP-49469149-136.243.18.41-1689171334293 heartbeating to localhost.localdomain/127.0.0.1:39107] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-49469149-136.243.18.41-1689171334293 (Datanode Uuid b3f92d14-88f4-4586-945a-eecfefe8b150) service to localhost.localdomain/127.0.0.1:39107 2023-07-12 14:16:04,920 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79/dfs/data/data1/current/BP-49469149-136.243.18.41-1689171334293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 14:16:04,921 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c127fcce-cf7b-606c-74a6-3163c9ac5425/cluster_70775bf0-b367-9c45-175c-824a780c6c79/dfs/data/data2/current/BP-49469149-136.243.18.41-1689171334293] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 14:16:04,956 INFO [Listener at localhost.localdomain/36341] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-07-12 14:16:05,074 INFO [Listener at localhost.localdomain/36341] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 14:16:05,130 INFO [Listener at localhost.localdomain/36341] hbase.HBaseTestingUtility(1293): Minicluster is down