2023-07-19 01:05:13,761 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-19 01:05:13,982 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f 2023-07-19 01:05:13,997 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-19 01:05:13,998 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3, deleteOnExit=true 2023-07-19 01:05:13,998 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-19 01:05:13,999 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/test.cache.data in system properties and HBase conf 2023-07-19 01:05:13,999 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/hadoop.tmp.dir in system properties and HBase conf 2023-07-19 01:05:14,000 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/hadoop.log.dir in system properties and HBase conf 2023-07-19 01:05:14,000 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-19 01:05:14,001 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-19 01:05:14,001 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-19 01:05:14,113 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-19 01:05:14,489 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-19 01:05:14,493 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-19 01:05:14,493 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-19 01:05:14,494 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-19 01:05:14,494 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-19 01:05:14,494 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-19 01:05:14,495 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-19 01:05:14,495 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-19 01:05:14,495 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-19 01:05:14,495 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-19 01:05:14,496 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/nfs.dump.dir in system properties and HBase conf 2023-07-19 01:05:14,496 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/java.io.tmpdir in system properties and HBase conf 2023-07-19 01:05:14,496 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-19 01:05:14,496 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-19 01:05:14,497 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-19 01:05:14,982 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-19 01:05:14,986 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-19 01:05:15,272 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-19 01:05:15,463 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-19 01:05:15,480 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 01:05:15,518 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 01:05:15,550 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/java.io.tmpdir/Jetty_localhost_42421_hdfs____.zeroyp/webapp 2023-07-19 01:05:15,716 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42421 2023-07-19 01:05:15,726 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-19 01:05:15,726 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-19 01:05:16,137 WARN [Listener at localhost/32877] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 01:05:16,244 WARN [Listener at localhost/32877] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-19 01:05:16,268 WARN [Listener at localhost/32877] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 01:05:16,275 INFO [Listener at localhost/32877] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 01:05:16,279 INFO [Listener at localhost/32877] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/java.io.tmpdir/Jetty_localhost_41959_datanode____1jpqzr/webapp 2023-07-19 01:05:16,379 INFO [Listener at localhost/32877] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41959 2023-07-19 01:05:16,699 WARN [Listener at localhost/36639] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 01:05:16,711 WARN [Listener at localhost/36639] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-19 01:05:16,714 WARN [Listener at localhost/36639] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 01:05:16,716 INFO [Listener at localhost/36639] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 01:05:16,721 INFO [Listener at localhost/36639] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/java.io.tmpdir/Jetty_localhost_46693_datanode____m9lfcl/webapp 2023-07-19 01:05:16,826 INFO [Listener at localhost/36639] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46693 2023-07-19 01:05:16,835 WARN [Listener at localhost/44241] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 01:05:17,180 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe1d28a8c6e56a5ba: Processing first storage report for DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93 from datanode c9b5721b-1253-49ab-b925-2d13db55f627 2023-07-19 01:05:17,182 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe1d28a8c6e56a5ba: from storage DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93 node DatanodeRegistration(127.0.0.1:38291, datanodeUuid=c9b5721b-1253-49ab-b925-2d13db55f627, infoPort=37021, infoSecurePort=0, ipcPort=36639, storageInfo=lv=-57;cid=testClusterID;nsid=168793545;c=1689728715060), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-19 01:05:17,182 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x55e1927866dc1de2: Processing first storage report for DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228 from datanode 57050ffe-2818-43d1-8aaa-eaa23fe4d1a5 2023-07-19 01:05:17,182 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x55e1927866dc1de2: from storage DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228 node DatanodeRegistration(127.0.0.1:45855, datanodeUuid=57050ffe-2818-43d1-8aaa-eaa23fe4d1a5, infoPort=36725, infoSecurePort=0, ipcPort=44241, storageInfo=lv=-57;cid=testClusterID;nsid=168793545;c=1689728715060), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 01:05:17,182 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe1d28a8c6e56a5ba: Processing first storage report for DS-68e0dff7-50a1-4e4a-97c7-878bef8fe925 from datanode c9b5721b-1253-49ab-b925-2d13db55f627 2023-07-19 01:05:17,182 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe1d28a8c6e56a5ba: from storage DS-68e0dff7-50a1-4e4a-97c7-878bef8fe925 node DatanodeRegistration(127.0.0.1:38291, datanodeUuid=c9b5721b-1253-49ab-b925-2d13db55f627, infoPort=37021, infoSecurePort=0, ipcPort=36639, storageInfo=lv=-57;cid=testClusterID;nsid=168793545;c=1689728715060), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 01:05:17,182 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x55e1927866dc1de2: Processing first storage report for DS-4ffd24bf-a235-489f-835b-01c5748e45d7 from datanode 57050ffe-2818-43d1-8aaa-eaa23fe4d1a5 2023-07-19 01:05:17,183 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x55e1927866dc1de2: from storage DS-4ffd24bf-a235-489f-835b-01c5748e45d7 node DatanodeRegistration(127.0.0.1:45855, datanodeUuid=57050ffe-2818-43d1-8aaa-eaa23fe4d1a5, infoPort=36725, infoSecurePort=0, ipcPort=44241, storageInfo=lv=-57;cid=testClusterID;nsid=168793545;c=1689728715060), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 01:05:17,240 DEBUG [Listener at localhost/44241] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f 2023-07-19 01:05:17,312 INFO [Listener at localhost/44241] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/zookeeper_0, clientPort=53242, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-19 01:05:17,329 INFO [Listener at localhost/44241] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=53242 2023-07-19 01:05:17,338 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:17,340 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:18,002 INFO [Listener at localhost/44241] util.FSUtils(471): Created version file at hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081 with version=8 2023-07-19 01:05:18,002 INFO [Listener at localhost/44241] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/hbase-staging 2023-07-19 01:05:18,311 INFO [Listener at localhost/44241] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-19 01:05:18,783 INFO [Listener at localhost/44241] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 01:05:18,815 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:18,815 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:18,816 INFO [Listener at localhost/44241] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 01:05:18,816 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:18,816 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 01:05:18,957 INFO [Listener at localhost/44241] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 01:05:19,028 DEBUG [Listener at localhost/44241] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-19 01:05:19,125 INFO [Listener at localhost/44241] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39087 2023-07-19 01:05:19,135 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:19,138 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:19,159 INFO [Listener at localhost/44241] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39087 connecting to ZooKeeper ensemble=127.0.0.1:53242 2023-07-19 01:05:19,199 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:390870x0, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 01:05:19,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:39087-0x1017b1c94f50000 connected 2023-07-19 01:05:19,227 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 01:05:19,228 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 01:05:19,232 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 01:05:19,239 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39087 2023-07-19 01:05:19,240 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39087 2023-07-19 01:05:19,240 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39087 2023-07-19 01:05:19,241 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39087 2023-07-19 01:05:19,241 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39087 2023-07-19 01:05:19,247 INFO [Listener at localhost/44241] master.HMaster(444): hbase.rootdir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081, hbase.cluster.distributed=false 2023-07-19 01:05:19,316 INFO [Listener at localhost/44241] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 01:05:19,317 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:19,317 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:19,317 INFO [Listener at localhost/44241] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 01:05:19,317 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:19,317 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 01:05:19,322 INFO [Listener at localhost/44241] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 01:05:19,325 INFO [Listener at localhost/44241] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45105 2023-07-19 01:05:19,327 INFO [Listener at localhost/44241] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 01:05:19,333 DEBUG [Listener at localhost/44241] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 01:05:19,335 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:19,337 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:19,339 INFO [Listener at localhost/44241] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45105 connecting to ZooKeeper ensemble=127.0.0.1:53242 2023-07-19 01:05:19,343 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:451050x0, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 01:05:19,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:45105-0x1017b1c94f50001 connected 2023-07-19 01:05:19,344 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 01:05:19,345 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 01:05:19,346 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 01:05:19,347 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45105 2023-07-19 01:05:19,347 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45105 2023-07-19 01:05:19,347 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45105 2023-07-19 01:05:19,348 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45105 2023-07-19 01:05:19,348 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45105 2023-07-19 01:05:19,362 INFO [Listener at localhost/44241] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 01:05:19,363 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:19,363 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:19,363 INFO [Listener at localhost/44241] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 01:05:19,364 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:19,364 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 01:05:19,364 INFO [Listener at localhost/44241] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 01:05:19,365 INFO [Listener at localhost/44241] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41065 2023-07-19 01:05:19,366 INFO [Listener at localhost/44241] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 01:05:19,367 DEBUG [Listener at localhost/44241] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 01:05:19,368 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:19,370 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:19,372 INFO [Listener at localhost/44241] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41065 connecting to ZooKeeper ensemble=127.0.0.1:53242 2023-07-19 01:05:19,377 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:410650x0, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 01:05:19,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:41065-0x1017b1c94f50002 connected 2023-07-19 01:05:19,378 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 01:05:19,379 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 01:05:19,380 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 01:05:19,380 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41065 2023-07-19 01:05:19,381 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41065 2023-07-19 01:05:19,381 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41065 2023-07-19 01:05:19,381 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41065 2023-07-19 01:05:19,382 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41065 2023-07-19 01:05:19,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:19,393 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-19 01:05:19,395 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:19,416 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 01:05:19,416 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 01:05:19,416 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 01:05:19,417 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:19,417 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 01:05:19,418 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39087,1689728718150 from backup master directory 2023-07-19 01:05:19,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 01:05:19,422 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:19,422 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-19 01:05:19,423 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 01:05:19,423 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:19,426 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-19 01:05:19,427 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-19 01:05:19,518 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/hbase.id with ID: f10690fa-cb6d-420e-abdc-d71279a7071e 2023-07-19 01:05:19,563 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:19,578 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:19,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1cc6ac18 to 127.0.0.1:53242 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 01:05:19,671 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20a1d0bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 01:05:19,696 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-19 01:05:19,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-19 01:05:19,718 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-19 01:05:19,718 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-19 01:05:19,720 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-19 01:05:19,725 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-19 01:05:19,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 01:05:19,763 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/data/master/store-tmp 2023-07-19 01:05:19,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:19,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-19 01:05:19,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 01:05:19,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 01:05:19,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-19 01:05:19,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 01:05:19,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 01:05:19,797 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 01:05:19,798 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/WALs/jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:19,820 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39087%2C1689728718150, suffix=, logDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/WALs/jenkins-hbase4.apache.org,39087,1689728718150, archiveDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/oldWALs, maxLogs=10 2023-07-19 01:05:19,877 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK] 2023-07-19 01:05:19,877 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK] 2023-07-19 01:05:19,886 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 01:05:19,953 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/WALs/jenkins-hbase4.apache.org,39087,1689728718150/jenkins-hbase4.apache.org%2C39087%2C1689728718150.1689728719830 2023-07-19 01:05:19,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK], DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK]] 2023-07-19 01:05:19,954 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:19,954 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:19,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 01:05:19,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 01:05:20,015 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-19 01:05:20,022 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-19 01:05:20,046 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-19 01:05:20,058 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:20,064 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-19 01:05:20,066 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-19 01:05:20,081 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 01:05:20,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 01:05:20,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67881595, jitterRate=0.011514589190483093}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 01:05:20,087 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 01:05:20,088 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-19 01:05:20,108 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-19 01:05:20,108 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-19 01:05:20,111 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-19 01:05:20,113 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-19 01:05:20,146 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-07-19 01:05:20,146 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-19 01:05:20,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-19 01:05:20,183 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-19 01:05:20,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-19 01:05:20,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-19 01:05:20,213 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-19 01:05:20,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-19 01:05:20,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-19 01:05:20,225 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:20,227 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-19 01:05:20,228 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-19 01:05:20,244 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-19 01:05:20,251 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 01:05:20,251 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 01:05:20,251 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 01:05:20,251 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:20,252 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,39087,1689728718150, sessionid=0x1017b1c94f50000, setting cluster-up flag (Was=false) 2023-07-19 01:05:20,271 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:20,277 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-19 01:05:20,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:20,285 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:20,293 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-19 01:05:20,294 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:20,297 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.hbase-snapshot/.tmp 2023-07-19 01:05:20,385 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(951): ClusterId : f10690fa-cb6d-420e-abdc-d71279a7071e 2023-07-19 01:05:20,385 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(951): ClusterId : f10690fa-cb6d-420e-abdc-d71279a7071e 2023-07-19 01:05:20,390 DEBUG [RS:0;jenkins-hbase4:45105] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 01:05:20,390 DEBUG [RS:1;jenkins-hbase4:41065] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 01:05:20,398 DEBUG [RS:0;jenkins-hbase4:45105] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 01:05:20,398 DEBUG [RS:1;jenkins-hbase4:41065] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 01:05:20,398 DEBUG [RS:0;jenkins-hbase4:45105] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 01:05:20,399 DEBUG [RS:1;jenkins-hbase4:41065] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 01:05:20,403 DEBUG [RS:0;jenkins-hbase4:45105] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 01:05:20,403 DEBUG [RS:1;jenkins-hbase4:41065] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 01:05:20,405 DEBUG [RS:0;jenkins-hbase4:45105] zookeeper.ReadOnlyZKClient(139): Connect 0x48048430 to 127.0.0.1:53242 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 01:05:20,406 DEBUG [RS:1;jenkins-hbase4:41065] zookeeper.ReadOnlyZKClient(139): Connect 0x6a796c69 to 127.0.0.1:53242 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 01:05:20,415 DEBUG [RS:1;jenkins-hbase4:41065] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64be8f56, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 01:05:20,416 DEBUG [RS:0;jenkins-hbase4:45105] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46c36340, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 01:05:20,416 DEBUG [RS:1;jenkins-hbase4:41065] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60810719, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 01:05:20,416 DEBUG [RS:0;jenkins-hbase4:45105] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42fcef40, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 01:05:20,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-19 01:05:20,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 01:05:20,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 01:05:20,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 01:05:20,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 01:05:20,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-19 01:05:20,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,438 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 01:05:20,438 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,440 DEBUG [RS:0;jenkins-hbase4:45105] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:45105 2023-07-19 01:05:20,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689728750442 2023-07-19 01:05:20,442 DEBUG [RS:1;jenkins-hbase4:41065] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:41065 2023-07-19 01:05:20,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-19 01:05:20,447 INFO [RS:1;jenkins-hbase4:41065] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 01:05:20,447 INFO [RS:1;jenkins-hbase4:41065] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 01:05:20,447 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-19 01:05:20,447 INFO [RS:0;jenkins-hbase4:45105] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 01:05:20,448 DEBUG [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 01:05:20,448 INFO [RS:0;jenkins-hbase4:45105] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 01:05:20,448 DEBUG [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 01:05:20,448 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-19 01:05:20,451 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,39087,1689728718150 with isa=jenkins-hbase4.apache.org/172.31.14.131:41065, startcode=1689728719362 2023-07-19 01:05:20,451 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,39087,1689728718150 with isa=jenkins-hbase4.apache.org/172.31.14.131:45105, startcode=1689728719316 2023-07-19 01:05:20,453 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-19 01:05:20,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-19 01:05:20,466 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-19 01:05:20,467 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-19 01:05:20,467 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-19 01:05:20,467 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-19 01:05:20,468 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,470 DEBUG [RS:1;jenkins-hbase4:41065] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 01:05:20,470 DEBUG [RS:0;jenkins-hbase4:45105] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 01:05:20,471 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-19 01:05:20,473 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-19 01:05:20,473 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-19 01:05:20,477 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-19 01:05:20,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-19 01:05:20,482 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689728720482,5,FailOnTimeoutGroup] 2023-07-19 01:05:20,484 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689728720483,5,FailOnTimeoutGroup] 2023-07-19 01:05:20,484 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,485 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-19 01:05:20,487 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,489 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,498 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-19 01:05:20,499 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-19 01:05:20,500 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081 2023-07-19 01:05:20,526 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:20,529 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 01:05:20,533 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/info 2023-07-19 01:05:20,534 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 01:05:20,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:20,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 01:05:20,539 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/rep_barrier 2023-07-19 01:05:20,540 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 01:05:20,541 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:20,542 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 01:05:20,542 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43163, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 01:05:20,542 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47243, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 01:05:20,545 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/table 2023-07-19 01:05:20,546 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 01:05:20,547 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:20,548 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740 2023-07-19 01:05:20,549 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740 2023-07-19 01:05:20,553 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 01:05:20,556 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 01:05:20,557 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39087] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:20,560 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 01:05:20,561 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74934776, jitterRate=0.1166151762008667}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 01:05:20,561 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 01:05:20,561 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 01:05:20,561 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 01:05:20,561 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 01:05:20,561 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 01:05:20,561 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 01:05:20,562 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 01:05:20,563 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 01:05:20,569 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39087] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:20,571 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-19 01:05:20,571 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-19 01:05:20,577 DEBUG [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081 2023-07-19 01:05:20,577 DEBUG [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081 2023-07-19 01:05:20,577 DEBUG [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:32877 2023-07-19 01:05:20,577 DEBUG [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:32877 2023-07-19 01:05:20,577 DEBUG [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 01:05:20,577 DEBUG [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 01:05:20,583 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-19 01:05:20,587 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:20,587 DEBUG [RS:0;jenkins-hbase4:45105] zookeeper.ZKUtil(162): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:20,587 DEBUG [RS:1;jenkins-hbase4:41065] zookeeper.ZKUtil(162): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:20,588 WARN [RS:0;jenkins-hbase4:45105] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 01:05:20,589 WARN [RS:1;jenkins-hbase4:41065] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 01:05:20,589 INFO [RS:0;jenkins-hbase4:45105] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 01:05:20,589 INFO [RS:1;jenkins-hbase4:41065] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 01:05:20,590 DEBUG [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1948): logDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:20,590 DEBUG [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1948): logDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:20,590 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,45105,1689728719316] 2023-07-19 01:05:20,590 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41065,1689728719362] 2023-07-19 01:05:20,600 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-19 01:05:20,602 DEBUG [RS:0;jenkins-hbase4:45105] zookeeper.ZKUtil(162): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:20,602 DEBUG [RS:1;jenkins-hbase4:41065] zookeeper.ZKUtil(162): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:20,603 DEBUG [RS:0;jenkins-hbase4:45105] zookeeper.ZKUtil(162): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:20,603 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-19 01:05:20,603 DEBUG [RS:1;jenkins-hbase4:41065] zookeeper.ZKUtil(162): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:20,614 DEBUG [RS:0;jenkins-hbase4:45105] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 01:05:20,614 DEBUG [RS:1;jenkins-hbase4:41065] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 01:05:20,624 INFO [RS:0;jenkins-hbase4:45105] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 01:05:20,624 INFO [RS:1;jenkins-hbase4:41065] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 01:05:20,644 INFO [RS:0;jenkins-hbase4:45105] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 01:05:20,644 INFO [RS:1;jenkins-hbase4:41065] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 01:05:20,648 INFO [RS:1;jenkins-hbase4:41065] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 01:05:20,648 INFO [RS:0;jenkins-hbase4:45105] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 01:05:20,649 INFO [RS:1;jenkins-hbase4:41065] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,649 INFO [RS:0;jenkins-hbase4:45105] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,650 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 01:05:20,650 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 01:05:20,658 INFO [RS:0;jenkins-hbase4:45105] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,658 INFO [RS:1;jenkins-hbase4:41065] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,658 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,658 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,658 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 01:05:20,659 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 01:05:20,659 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,659 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,660 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,660 DEBUG [RS:0;jenkins-hbase4:45105] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,660 DEBUG [RS:1;jenkins-hbase4:41065] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:20,661 INFO [RS:0;jenkins-hbase4:45105] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,661 INFO [RS:1;jenkins-hbase4:41065] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,661 INFO [RS:0;jenkins-hbase4:45105] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,661 INFO [RS:1;jenkins-hbase4:41065] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,661 INFO [RS:0;jenkins-hbase4:45105] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,661 INFO [RS:1;jenkins-hbase4:41065] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,678 INFO [RS:0;jenkins-hbase4:45105] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 01:05:20,678 INFO [RS:1;jenkins-hbase4:41065] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 01:05:20,681 INFO [RS:1;jenkins-hbase4:41065] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41065,1689728719362-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,681 INFO [RS:0;jenkins-hbase4:45105] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45105,1689728719316-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:20,702 INFO [RS:0;jenkins-hbase4:45105] regionserver.Replication(203): jenkins-hbase4.apache.org,45105,1689728719316 started 2023-07-19 01:05:20,703 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,45105,1689728719316, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:45105, sessionid=0x1017b1c94f50001 2023-07-19 01:05:20,703 INFO [RS:1;jenkins-hbase4:41065] regionserver.Replication(203): jenkins-hbase4.apache.org,41065,1689728719362 started 2023-07-19 01:05:20,703 DEBUG [RS:0;jenkins-hbase4:45105] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 01:05:20,703 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,41065,1689728719362, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41065, sessionid=0x1017b1c94f50002 2023-07-19 01:05:20,703 DEBUG [RS:0;jenkins-hbase4:45105] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:20,703 DEBUG [RS:1;jenkins-hbase4:41065] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 01:05:20,704 DEBUG [RS:0;jenkins-hbase4:45105] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45105,1689728719316' 2023-07-19 01:05:20,704 DEBUG [RS:1;jenkins-hbase4:41065] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:20,704 DEBUG [RS:0;jenkins-hbase4:45105] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 01:05:20,704 DEBUG [RS:1;jenkins-hbase4:41065] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41065,1689728719362' 2023-07-19 01:05:20,705 DEBUG [RS:1;jenkins-hbase4:41065] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 01:05:20,705 DEBUG [RS:0;jenkins-hbase4:45105] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 01:05:20,705 DEBUG [RS:1;jenkins-hbase4:41065] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 01:05:20,706 DEBUG [RS:0;jenkins-hbase4:45105] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 01:05:20,706 DEBUG [RS:1;jenkins-hbase4:41065] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 01:05:20,706 DEBUG [RS:0;jenkins-hbase4:45105] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 01:05:20,706 DEBUG [RS:1;jenkins-hbase4:41065] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 01:05:20,706 DEBUG [RS:0;jenkins-hbase4:45105] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:20,706 DEBUG [RS:1;jenkins-hbase4:41065] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:20,707 DEBUG [RS:1;jenkins-hbase4:41065] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41065,1689728719362' 2023-07-19 01:05:20,707 DEBUG [RS:1;jenkins-hbase4:41065] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 01:05:20,707 DEBUG [RS:0;jenkins-hbase4:45105] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45105,1689728719316' 2023-07-19 01:05:20,707 DEBUG [RS:0;jenkins-hbase4:45105] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 01:05:20,707 DEBUG [RS:1;jenkins-hbase4:41065] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 01:05:20,708 DEBUG [RS:0;jenkins-hbase4:45105] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 01:05:20,708 DEBUG [RS:1;jenkins-hbase4:41065] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 01:05:20,708 DEBUG [RS:0;jenkins-hbase4:45105] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 01:05:20,708 INFO [RS:1;jenkins-hbase4:41065] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 01:05:20,708 INFO [RS:0;jenkins-hbase4:45105] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 01:05:20,709 INFO [RS:1;jenkins-hbase4:41065] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 01:05:20,709 INFO [RS:0;jenkins-hbase4:45105] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 01:05:20,755 DEBUG [jenkins-hbase4:39087] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-19 01:05:20,759 DEBUG [jenkins-hbase4:39087] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 01:05:20,765 DEBUG [jenkins-hbase4:39087] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 01:05:20,765 DEBUG [jenkins-hbase4:39087] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 01:05:20,765 DEBUG [jenkins-hbase4:39087] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 01:05:20,768 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45105,1689728719316, state=OPENING 2023-07-19 01:05:20,775 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-19 01:05:20,777 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:20,777 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 01:05:20,782 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45105,1689728719316}] 2023-07-19 01:05:20,820 INFO [RS:0;jenkins-hbase4:45105] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45105%2C1689728719316, suffix=, logDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316, archiveDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs, maxLogs=32 2023-07-19 01:05:20,820 INFO [RS:1;jenkins-hbase4:41065] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41065%2C1689728719362, suffix=, logDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362, archiveDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs, maxLogs=32 2023-07-19 01:05:20,844 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK] 2023-07-19 01:05:20,845 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK] 2023-07-19 01:05:20,845 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK] 2023-07-19 01:05:20,845 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK] 2023-07-19 01:05:20,852 INFO [RS:1;jenkins-hbase4:41065] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824 2023-07-19 01:05:20,852 INFO [RS:0;jenkins-hbase4:45105] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824 2023-07-19 01:05:20,854 DEBUG [RS:1;jenkins-hbase4:41065] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK], DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK]] 2023-07-19 01:05:20,854 DEBUG [RS:0;jenkins-hbase4:45105] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK], DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK]] 2023-07-19 01:05:20,967 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:20,969 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 01:05:20,973 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51922, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 01:05:20,987 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-19 01:05:20,987 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 01:05:20,990 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta, suffix=.meta, logDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316, archiveDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs, maxLogs=32 2023-07-19 01:05:21,009 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK] 2023-07-19 01:05:21,009 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK] 2023-07-19 01:05:21,016 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta 2023-07-19 01:05:21,018 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK], DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK]] 2023-07-19 01:05:21,018 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:21,021 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-19 01:05:21,044 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-19 01:05:21,085 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-19 01:05:21,090 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-19 01:05:21,091 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:21,091 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-19 01:05:21,091 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-19 01:05:21,094 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 01:05:21,096 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/info 2023-07-19 01:05:21,096 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/info 2023-07-19 01:05:21,097 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 01:05:21,098 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:21,098 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 01:05:21,100 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/rep_barrier 2023-07-19 01:05:21,100 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/rep_barrier 2023-07-19 01:05:21,100 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 01:05:21,101 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:21,101 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 01:05:21,103 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/table 2023-07-19 01:05:21,103 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/table 2023-07-19 01:05:21,104 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 01:05:21,104 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:21,106 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740 2023-07-19 01:05:21,109 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740 2023-07-19 01:05:21,112 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 01:05:21,115 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 01:05:21,116 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60768713, jitterRate=-0.09447561204433441}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 01:05:21,116 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 01:05:21,128 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689728720959 2023-07-19 01:05:21,146 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-19 01:05:21,147 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-19 01:05:21,147 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45105,1689728719316, state=OPEN 2023-07-19 01:05:21,150 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 01:05:21,150 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 01:05:21,155 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-19 01:05:21,155 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45105,1689728719316 in 368 msec 2023-07-19 01:05:21,160 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-19 01:05:21,160 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 573 msec 2023-07-19 01:05:21,165 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 812 msec 2023-07-19 01:05:21,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689728721165, completionTime=-1 2023-07-19 01:05:21,166 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-19 01:05:21,167 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-19 01:05:21,236 DEBUG [hconnection-0x70dfd7d3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:21,240 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51938, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:21,257 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-19 01:05:21,258 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689728781258 2023-07-19 01:05:21,258 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689728841258 2023-07-19 01:05:21,258 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 91 msec 2023-07-19 01:05:21,282 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39087,1689728718150-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:21,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39087,1689728718150-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:21,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39087,1689728718150-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:21,285 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39087, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:21,285 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:21,293 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-19 01:05:21,302 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-19 01:05:21,304 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-19 01:05:21,314 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-19 01:05:21,316 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-19 01:05:21,319 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-19 01:05:21,345 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:21,348 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772 empty. 2023-07-19 01:05:21,349 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:21,349 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-19 01:05:21,376 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-19 01:05:21,379 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => d8f2f6dea51dd237660eba5bde311772, NAME => 'hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp 2023-07-19 01:05:21,397 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:21,397 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing d8f2f6dea51dd237660eba5bde311772, disabling compactions & flushes 2023-07-19 01:05:21,397 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:21,397 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:21,397 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. after waiting 0 ms 2023-07-19 01:05:21,397 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:21,397 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:21,397 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for d8f2f6dea51dd237660eba5bde311772: 2023-07-19 01:05:21,402 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-19 01:05:21,420 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689728721408"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689728721408"}]},"ts":"1689728721408"} 2023-07-19 01:05:21,450 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-19 01:05:21,452 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-19 01:05:21,457 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689728721452"}]},"ts":"1689728721452"} 2023-07-19 01:05:21,462 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-19 01:05:21,466 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 01:05:21,468 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 01:05:21,468 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 01:05:21,468 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 01:05:21,470 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN}] 2023-07-19 01:05:21,473 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN 2023-07-19 01:05:21,476 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41065,1689728719362; forceNewPlan=false, retain=false 2023-07-19 01:05:21,628 INFO [jenkins-hbase4:39087] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-19 01:05:21,629 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d8f2f6dea51dd237660eba5bde311772, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:21,630 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689728721629"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689728721629"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689728721629"}]},"ts":"1689728721629"} 2023-07-19 01:05:21,634 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure d8f2f6dea51dd237660eba5bde311772, server=jenkins-hbase4.apache.org,41065,1689728719362}] 2023-07-19 01:05:21,788 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:21,788 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 01:05:21,792 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40518, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 01:05:21,798 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:21,798 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d8f2f6dea51dd237660eba5bde311772, NAME => 'hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:21,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:21,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:21,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:21,800 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:21,802 INFO [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:21,805 DEBUG [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/info 2023-07-19 01:05:21,805 DEBUG [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/info 2023-07-19 01:05:21,806 INFO [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d8f2f6dea51dd237660eba5bde311772 columnFamilyName info 2023-07-19 01:05:21,807 INFO [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] regionserver.HStore(310): Store=d8f2f6dea51dd237660eba5bde311772/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:21,809 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:21,810 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:21,815 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:21,820 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 01:05:21,821 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d8f2f6dea51dd237660eba5bde311772; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60737445, jitterRate=-0.09494154155254364}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 01:05:21,821 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d8f2f6dea51dd237660eba5bde311772: 2023-07-19 01:05:21,824 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772., pid=6, masterSystemTime=1689728721788 2023-07-19 01:05:21,829 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:21,829 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:21,831 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d8f2f6dea51dd237660eba5bde311772, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:21,831 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689728721830"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689728721830"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689728721830"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689728721830"}]},"ts":"1689728721830"} 2023-07-19 01:05:21,840 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-19 01:05:21,840 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure d8f2f6dea51dd237660eba5bde311772, server=jenkins-hbase4.apache.org,41065,1689728719362 in 203 msec 2023-07-19 01:05:21,844 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-19 01:05:21,845 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN in 370 msec 2023-07-19 01:05:21,846 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-19 01:05:21,846 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689728721846"}]},"ts":"1689728721846"} 2023-07-19 01:05:21,849 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-19 01:05:21,853 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-19 01:05:21,856 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 549 msec 2023-07-19 01:05:21,917 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-19 01:05:21,918 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-19 01:05:21,918 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:21,941 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:21,943 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40528, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:21,959 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-19 01:05:21,985 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-19 01:05:21,993 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 42 msec 2023-07-19 01:05:22,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-19 01:05:22,017 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-19 01:05:22,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 19 msec 2023-07-19 01:05:22,032 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-19 01:05:22,036 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-19 01:05:22,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.613sec 2023-07-19 01:05:22,041 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-19 01:05:22,043 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-19 01:05:22,043 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-19 01:05:22,045 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39087,1689728718150-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-19 01:05:22,045 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39087,1689728718150-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-19 01:05:22,057 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-19 01:05:22,092 DEBUG [Listener at localhost/44241] zookeeper.ReadOnlyZKClient(139): Connect 0x5f87e9f1 to 127.0.0.1:53242 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 01:05:22,097 DEBUG [Listener at localhost/44241] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5363c3cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 01:05:22,111 DEBUG [hconnection-0x3a30b9dc-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:22,124 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51950, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:22,135 INFO [Listener at localhost/44241] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:22,178 DEBUG [Listener at localhost/44241] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-19 01:05:22,181 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-19 01:05:22,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-19 01:05:22,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-19 01:05:22,201 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-19 01:05:22,203 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-19 01:05:22,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-19 01:05:22,206 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp/data/default/t/9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,207 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp/data/default/t/9a00fc7c9da6b8edd769d227741a103a empty. 2023-07-19 01:05:22,209 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp/data/default/t/9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,209 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-19 01:05:22,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 01:05:22,231 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-19 01:05:22,233 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9a00fc7c9da6b8edd769d227741a103a, NAME => 't,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/.tmp 2023-07-19 01:05:22,251 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:22,251 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 9a00fc7c9da6b8edd769d227741a103a, disabling compactions & flushes 2023-07-19 01:05:22,251 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,251 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,251 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. after waiting 0 ms 2023-07-19 01:05:22,251 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,251 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,251 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 9a00fc7c9da6b8edd769d227741a103a: 2023-07-19 01:05:22,256 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-19 01:05:22,257 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689728722257"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689728722257"}]},"ts":"1689728722257"} 2023-07-19 01:05:22,260 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-19 01:05:22,262 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-19 01:05:22,262 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689728722262"}]},"ts":"1689728722262"} 2023-07-19 01:05:22,264 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-19 01:05:22,270 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 01:05:22,271 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 01:05:22,271 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 01:05:22,271 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 01:05:22,272 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN}] 2023-07-19 01:05:22,274 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN 2023-07-19 01:05:22,276 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41065,1689728719362; forceNewPlan=false, retain=false 2023-07-19 01:05:22,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 01:05:22,426 INFO [jenkins-hbase4:39087] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-19 01:05:22,427 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=9a00fc7c9da6b8edd769d227741a103a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:22,427 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689728722427"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689728722427"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689728722427"}]},"ts":"1689728722427"} 2023-07-19 01:05:22,431 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 9a00fc7c9da6b8edd769d227741a103a, server=jenkins-hbase4.apache.org,41065,1689728719362}] 2023-07-19 01:05:22,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 01:05:22,590 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9a00fc7c9da6b8edd769d227741a103a, NAME => 't,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:22,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:22,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,593 INFO [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,596 DEBUG [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/f 2023-07-19 01:05:22,596 DEBUG [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/f 2023-07-19 01:05:22,596 INFO [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a00fc7c9da6b8edd769d227741a103a columnFamilyName f 2023-07-19 01:05:22,598 INFO [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] regionserver.HStore(310): Store=9a00fc7c9da6b8edd769d227741a103a/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:22,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 01:05:22,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9a00fc7c9da6b8edd769d227741a103a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70680869, jitterRate=0.0532270222902298}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 01:05:22,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9a00fc7c9da6b8edd769d227741a103a: 2023-07-19 01:05:22,610 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a., pid=11, masterSystemTime=1689728722584 2023-07-19 01:05:22,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,614 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=9a00fc7c9da6b8edd769d227741a103a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:22,615 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689728722614"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689728722614"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689728722614"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689728722614"}]},"ts":"1689728722614"} 2023-07-19 01:05:22,621 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-19 01:05:22,621 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 9a00fc7c9da6b8edd769d227741a103a, server=jenkins-hbase4.apache.org,41065,1689728719362 in 187 msec 2023-07-19 01:05:22,625 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-19 01:05:22,625 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN in 350 msec 2023-07-19 01:05:22,626 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-19 01:05:22,626 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689728722626"}]},"ts":"1689728722626"} 2023-07-19 01:05:22,629 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-19 01:05:22,632 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-19 01:05:22,635 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 436 msec 2023-07-19 01:05:22,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 01:05:22,826 INFO [Listener at localhost/44241] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-19 01:05:22,826 DEBUG [Listener at localhost/44241] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-19 01:05:22,828 INFO [Listener at localhost/44241] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-19 01:05:22,832 INFO [Listener at localhost/44241] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-19 01:05:22,832 INFO [Listener at localhost/44241] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-19 01:05:22,833 INFO [Listener at localhost/44241] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-19 01:05:22,836 DEBUG [Listener at localhost/44241] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:22,839 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40532, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:22,884 INFO [Listener at localhost/44241] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=295, OpenFileDescriptor=587, MaxFileDescriptor=60000, SystemLoadAverage=148, ProcessCount=171, AvailableMemoryMB=5889 2023-07-19 01:05:22,886 INFO [Listener at localhost/44241] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-19 01:05:22,890 DEBUG [Listener at localhost/44241] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 01:05:22,891 DEBUG [Listener at localhost/44241] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,891 INFO [Listener at localhost/44241] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-19 01:05:22,891 INFO [Listener at localhost/44241] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-19 01:05:22,894 INFO [Listener at localhost/44241] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689728721846/Put/vlen=2/seqid=0} 2023-07-19 01:05:22,895 INFO [Listener at localhost/44241] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-19 01:05:22,895 INFO [Listener at localhost/44241] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772./info:regioninfo/1689728721830/Put/vlen=41/seqid=0, hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772./info:seqnumDuringOpen/1689728721830/Put/vlen=8/seqid=0, hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772./info:server/1689728721830/Put/vlen=31/seqid=0, hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772./info:serverstartcode/1689728721830/Put/vlen=8/seqid=0, hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772./info:sn/1689728721629/Put/vlen=45/seqid=0, hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772./info:state/1689728721830/Put/vlen=4/seqid=0} 2023-07-19 01:05:22,895 INFO [Listener at localhost/44241] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => d8f2f6dea51dd237660eba5bde311772, NAME => 'hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:22,895 INFO [Listener at localhost/44241] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689728722626/Put/vlen=2/seqid=0} 2023-07-19 01:05:22,895 INFO [Listener at localhost/44241] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-19 01:05:22,895 INFO [Listener at localhost/44241] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a./info:regioninfo/1689728722614/Put/vlen=35/seqid=0, t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a./info:seqnumDuringOpen/1689728722614/Put/vlen=8/seqid=0, t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a./info:server/1689728722614/Put/vlen=31/seqid=0, t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a./info:serverstartcode/1689728722614/Put/vlen=8/seqid=0, t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a./info:sn/1689728722427/Put/vlen=45/seqid=0, t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a./info:state/1689728722614/Put/vlen=4/seqid=0} 2023-07-19 01:05:22,896 INFO [Listener at localhost/44241] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 9a00fc7c9da6b8edd769d227741a103a, NAME => 't,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:22,897 DEBUG [Listener at localhost/44241] zookeeper.ReadOnlyZKClient(139): Connect 0x28d47ff0 to 127.0.0.1:53242 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 01:05:22,905 DEBUG [Listener at localhost/44241] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4147d733, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 01:05:22,905 INFO [Listener at localhost/44241] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-19 01:05:22,905 INFO [Listener at localhost/44241] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-19 01:05:22,908 DEBUG [hconnection-0x1ef800db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:22,911 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51956, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:22,915 DEBUG [hconnection-0x1ef800db-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:22,918 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40546, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:22,921 INFO [Listener at localhost/44241] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-19 01:05:22,924 ERROR [Listener at localhost/44241] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,41065,1689728719362: die! ***** 2023-07-19 01:05:22,925 ERROR [Listener at localhost/44241] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-19 01:05:22,928 DEBUG [Listener at localhost/44241] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-19 01:05:22,931 DEBUG [Listener at localhost/44241] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-19 01:05:22,931 DEBUG [Listener at localhost/44241] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-19 01:05:22,932 DEBUG [Listener at localhost/44241] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-19 01:05:22,932 INFO [Listener at localhost/44241] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 968884224, "init": 513802240, "max": 2051014656, "used": 166773240 }, "NonHeapMemoryUsage": { "committed": 113139712, "init": 2555904, "max": -1, "used": 111170032 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-19 01:05:22,936 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39087] master.MasterRpcServices(612): jenkins-hbase4.apache.org,41065,1689728719362 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,41065,1689728719362: die! ***** 2023-07-19 01:05:22,939 INFO [Listener at localhost/44241] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,41065,1689728719362' ***** 2023-07-19 01:05:22,940 INFO [Listener at localhost/44241] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-19 01:05:22,940 INFO [RS:1;jenkins-hbase4:41065] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 01:05:22,940 INFO [RS:1;jenkins-hbase4:41065] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-19 01:05:22,940 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 01:05:22,940 INFO [RS:1;jenkins-hbase4:41065] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-19 01:05:22,941 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(3305): Received CLOSE for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:22,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41065] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -228316441365970942 2023-07-19 01:05:22,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41065] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41065,1689728719362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 01:05:22,942 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(3305): Received CLOSE for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d8f2f6dea51dd237660eba5bde311772, disabling compactions & flushes 2023-07-19 01:05:22,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41065] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:40546 deadline: 1689728782941, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41065,1689728719362 aborting 2023-07-19 01:05:22,943 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:22,943 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:22,943 DEBUG [RS:1;jenkins-hbase4:41065] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a796c69 to 127.0.0.1:53242 2023-07-19 01:05:22,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:22,943 DEBUG [RS:1;jenkins-hbase4:41065] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:22,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. after waiting 0 ms 2023-07-19 01:05:22,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:22,943 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-19 01:05:22,944 DEBUG [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1478): Online Regions={d8f2f6dea51dd237660eba5bde311772=hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772., 9a00fc7c9da6b8edd769d227741a103a=t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.} 2023-07-19 01:05:22,944 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:22,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:22,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d8f2f6dea51dd237660eba5bde311772: 2023-07-19 01:05:22,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:22,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9a00fc7c9da6b8edd769d227741a103a, disabling compactions & flushes 2023-07-19 01:05:22,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. after waiting 0 ms 2023-07-19 01:05:22,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,947 DEBUG [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1504): Waiting on 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:22,947 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,950 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9a00fc7c9da6b8edd769d227741a103a: 2023-07-19 01:05:22,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:22,965 DEBUG [Listener at localhost/44241] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41065,1689728719362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41065,1689728719362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-19 01:05:22,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41065] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41065,1689728719362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 01:05:22,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41065] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:40546 deadline: 1689728782969, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41065,1689728719362 aborting 2023-07-19 01:05:23,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41065] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41065,1689728719362 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 01:05:23,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41065] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:40546 deadline: 1689728783078, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41065,1689728719362 aborting 2023-07-19 01:05:23,150 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,41065,1689728719362; all regions closed. 2023-07-19 01:05:23,161 DEBUG [RS:1;jenkins-hbase4:41065] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:23,161 INFO [RS:1;jenkins-hbase4:41065] regionserver.LeaseManager(133): Closed leases 2023-07-19 01:05:23,161 INFO [RS:1;jenkins-hbase4:41065] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-19 01:05:23,162 INFO [RS:1;jenkins-hbase4:41065] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 01:05:23,162 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 01:05:23,162 INFO [RS:1;jenkins-hbase4:41065] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 01:05:23,162 INFO [RS:1;jenkins-hbase4:41065] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 01:05:23,163 INFO [RS:1;jenkins-hbase4:41065] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41065 2023-07-19 01:05:23,172 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:23,172 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:23,172 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:23,172 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:23,172 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:23,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:23,174 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41065,1689728719362] 2023-07-19 01:05:23,175 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,41065,1689728719362 znode expired, triggering replicatorRemoved event 2023-07-19 01:05:23,175 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41065,1689728719362; numProcessing=1 2023-07-19 01:05:23,178 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41065,1689728719362 already deleted, retry=false 2023-07-19 01:05:23,178 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,41065,1689728719362 on jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:23,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:23,185 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,41065,1689728719362, splitWal=true, meta=false 2023-07-19 01:05:23,185 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,41065,1689728719362 (carryingMeta=false) jenkins-hbase4.apache.org,41065,1689728719362/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1e0ad8f5[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-19 01:05:23,188 INFO [PEWorker-3] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41065,1689728719362, splitWal=true, meta=false 2023-07-19 01:05:23,190 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,41065,1689728719362 had 2 regions 2023-07-19 01:05:23,192 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41065,1689728719362, splitWal=true, meta=false, isMeta: false 2023-07-19 01:05:23,194 DEBUG [PEWorker-3] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting 2023-07-19 01:05:23,195 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,41065,1689728719362 WAL count=1, meta=false 2023-07-19 01:05:23,199 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824}] 2023-07-19 01:05:23,207 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:23,208 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824, worker=jenkins-hbase4.apache.org,45105,1689728719316}] 2023-07-19 01:05:23,275 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 01:05:23,275 INFO [RS:1;jenkins-hbase4:41065] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,41065,1689728719362; zookeeper connection closed. 2023-07-19 01:05:23,275 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:41065-0x1017b1c94f50002, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 01:05:23,276 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4a02981b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4a02981b 2023-07-19 01:05:23,286 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:41065 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41065 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 01:05:23,288 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:41065 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41065 2023-07-19 01:05:23,373 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45105] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-19 01:05:23,395 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824, size=1.6 K (1665bytes) 2023-07-19 01:05:23,396 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824 2023-07-19 01:05:23,401 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824 after 5ms 2023-07-19 01:05:23,406 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824 took 13ms 2023-07-19 01:05:23,422 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for d8f2f6dea51dd237660eba5bde311772: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-19 01:05:23,427 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 9a00fc7c9da6b8edd769d227741a103a: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-19 01:05:23,430 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824 so closing down 2023-07-19 01:05:23,430 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 01:05:23,435 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-0] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-19 01:05:23,442 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp 2023-07-19 01:05:23,443 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-0] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp 2023-07-19 01:05:23,445 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 01:05:23,480 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-19 01:05:23,483 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000005 2023-07-19 01:05:23,487 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-19 01:05:23,489 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000013 2023-07-19 01:05:23,490 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 82 ms; skipped=2; WAL=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824, size=1.6 K, length=1665, corrupted=false, cancelled=false 2023-07-19 01:05:23,490 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824, journal: Splitting hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824, size=1.6 K (1665bytes) at 1689728723395Finishing writing output for hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824 so closing down at 1689728723430 (+35 ms)Creating recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp at 1689728723442 (+12 ms)Creating recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp at 1689728723444 (+2 ms)3 split writer threads finished at 1689728723445 (+1 ms)Closed recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689728723480 (+35 ms)Rename recovered edits hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000005 at 1689728723483 (+3 ms)Closed recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689728723487 (+4 ms)Rename recovered edits hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824.temp to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000013 at 1689728723489 (+2 ms)Processed 14 edits across 2 Regions in 82 ms; skipped=2; WAL=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824, size=1.6 K, length=1665, corrupted=false, cancelled=false at 1689728723490 (+1 ms) 2023-07-19 01:05:23,491 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824 2023-07-19 01:05:23,494 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-19 01:05:23,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] master.HMaster(3641): Remote procedure done, pid=14 2023-07-19 01:05:23,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] wal.WALSplitUtil(141): Moved hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting/jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824 to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs 2023-07-19 01:05:23,509 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-19 01:05:23,509 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824, worker=jenkins-hbase4.apache.org,45105,1689728719316 in 298 msec 2023-07-19 01:05:23,511 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:23,515 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-19 01:05:23,515 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C41065%2C1689728719362.1689728720824, worker=jenkins-hbase4.apache.org,45105,1689728719316 in 314 msec 2023-07-19 01:05:23,517 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting dir is empty, no logs to split. 2023-07-19 01:05:23,517 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,41065,1689728719362 WAL count=0, meta=false 2023-07-19 01:05:23,517 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,41065,1689728719362 WAL splitting is done? wals=0, meta=false 2023-07-19 01:05:23,521 INFO [PEWorker-3] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,41065,1689728719362 failed, ignore...File hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,41065,1689728719362-splitting does not exist. 2023-07-19 01:05:23,523 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN}] 2023-07-19 01:05:23,524 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN 2023-07-19 01:05:23,524 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN 2023-07-19 01:05:23,526 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 01:05:23,526 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 01:05:23,595 DEBUG [hconnection-0x1ef800db-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41065 this server is in the failed servers list 2023-07-19 01:05:23,663 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 01:05:23,676 DEBUG [jenkins-hbase4:39087] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 01:05:23,676 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=d8f2f6dea51dd237660eba5bde311772, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:23,676 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=9a00fc7c9da6b8edd769d227741a103a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:23,677 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689728723676"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689728723676"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689728723676"}]},"ts":"1689728723676"} 2023-07-19 01:05:23,677 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689728723676"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689728723676"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689728723676"}]},"ts":"1689728723676"} 2023-07-19 01:05:23,681 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure 9a00fc7c9da6b8edd769d227741a103a, server=jenkins-hbase4.apache.org,45105,1689728719316}] 2023-07-19 01:05:23,682 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=16, state=RUNNABLE; OpenRegionProcedure d8f2f6dea51dd237660eba5bde311772, server=jenkins-hbase4.apache.org,45105,1689728719316}] 2023-07-19 01:05:23,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:23,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9a00fc7c9da6b8edd769d227741a103a, NAME => 't,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:23,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:23,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:23,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:23,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:23,843 INFO [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:23,845 DEBUG [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/f 2023-07-19 01:05:23,845 DEBUG [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/f 2023-07-19 01:05:23,845 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:23,846 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d8f2f6dea51dd237660eba5bde311772, NAME => 'hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:23,846 INFO [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a00fc7c9da6b8edd769d227741a103a columnFamilyName f 2023-07-19 01:05:23,846 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:23,846 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:23,846 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:23,846 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:23,847 INFO [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] regionserver.HStore(310): Store=9a00fc7c9da6b8edd769d227741a103a/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:23,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:23,849 INFO [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:23,851 DEBUG [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/info 2023-07-19 01:05:23,851 DEBUG [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/info 2023-07-19 01:05:23,851 INFO [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d8f2f6dea51dd237660eba5bde311772 columnFamilyName info 2023-07-19 01:05:23,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:23,852 INFO [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] regionserver.HStore(310): Store=d8f2f6dea51dd237660eba5bde311772/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:23,854 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000013 2023-07-19 01:05:23,854 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:23,857 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:23,858 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000005 2023-07-19 01:05:23,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000013 2023-07-19 01:05:23,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9a00fc7c9da6b8edd769d227741a103a 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-19 01:05:23,863 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000005 2023-07-19 01:05:23,863 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d8f2f6dea51dd237660eba5bde311772 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-19 01:05:23,921 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/.tmp/info/11e7afd179ba466e9876ce45532063f2 2023-07-19 01:05:23,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/.tmp/f/d98835a4e6e24223b5b213d5024b152b 2023-07-19 01:05:23,966 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/.tmp/info/11e7afd179ba466e9876ce45532063f2 as hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/info/11e7afd179ba466e9876ce45532063f2 2023-07-19 01:05:23,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/.tmp/f/d98835a4e6e24223b5b213d5024b152b as hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/f/d98835a4e6e24223b5b213d5024b152b 2023-07-19 01:05:23,975 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/info/11e7afd179ba466e9876ce45532063f2, entries=2, sequenceid=5, filesize=4.8 K 2023-07-19 01:05:23,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/f/d98835a4e6e24223b5b213d5024b152b, entries=10, sequenceid=13, filesize=4.8 K 2023-07-19 01:05:23,977 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for d8f2f6dea51dd237660eba5bde311772 in 114ms, sequenceid=5, compaction requested=false; wal=null 2023-07-19 01:05:23,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 9a00fc7c9da6b8edd769d227741a103a in 115ms, sequenceid=13, compaction requested=false; wal=null 2023-07-19 01:05:23,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-19 01:05:23,978 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-19 01:05:23,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/0000000000000000013 2023-07-19 01:05:23,982 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/0000000000000000005 2023-07-19 01:05:23,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:23,988 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:23,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-19 01:05:23,993 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-19 01:05:23,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9a00fc7c9da6b8edd769d227741a103a; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62693340, jitterRate=-0.06579643487930298}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 01:05:23,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9a00fc7c9da6b8edd769d227741a103a: 2023-07-19 01:05:23,995 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d8f2f6dea51dd237660eba5bde311772; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58847135, jitterRate=-0.1231093555688858}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 01:05:23,995 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d8f2f6dea51dd237660eba5bde311772: 2023-07-19 01:05:23,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a., pid=17, masterSystemTime=1689728723835 2023-07-19 01:05:23,996 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772., pid=18, masterSystemTime=1689728723835 2023-07-19 01:05:23,999 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:23,999 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:23,999 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=d8f2f6dea51dd237660eba5bde311772, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:24,000 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689728723999"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689728723999"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689728723999"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689728723999"}]},"ts":"1689728723999"} 2023-07-19 01:05:24,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:24,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:24,001 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=9a00fc7c9da6b8edd769d227741a103a, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:24,002 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689728724000"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689728724000"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689728724000"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689728724000"}]},"ts":"1689728724000"} 2023-07-19 01:05:24,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=16 2023-07-19 01:05:24,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=16, state=SUCCESS; OpenRegionProcedure d8f2f6dea51dd237660eba5bde311772, server=jenkins-hbase4.apache.org,45105,1689728719316 in 321 msec 2023-07-19 01:05:24,008 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=15 2023-07-19 01:05:24,008 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure 9a00fc7c9da6b8edd769d227741a103a, server=jenkins-hbase4.apache.org,45105,1689728719316 in 324 msec 2023-07-19 01:05:24,009 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN in 484 msec 2023-07-19 01:05:24,011 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-19 01:05:24,011 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,41065,1689728719362 after splitting done 2023-07-19 01:05:24,011 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN in 485 msec 2023-07-19 01:05:24,011 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,41065,1689728719362 from processing; numProcessing=0 2023-07-19 01:05:24,014 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,41065,1689728719362, splitWal=true, meta=false in 832 msec 2023-07-19 01:05:24,120 DEBUG [Listener at localhost/44241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28d47ff0 to 127.0.0.1:53242 2023-07-19 01:05:24,120 DEBUG [Listener at localhost/44241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:24,120 INFO [Listener at localhost/44241] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-19 01:05:24,134 INFO [Listener at localhost/44241] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=272 (was 295), OpenFileDescriptor=575 (was 587), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=153 (was 148) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=5880 (was 5889) 2023-07-19 01:05:24,146 INFO [Listener at localhost/44241] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=272, OpenFileDescriptor=575, MaxFileDescriptor=60000, SystemLoadAverage=153, ProcessCount=171, AvailableMemoryMB=5879 2023-07-19 01:05:24,146 INFO [Listener at localhost/44241] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:41065 2023-07-19 01:05:24,158 INFO [Listener at localhost/44241] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 01:05:24,158 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:24,158 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:24,158 INFO [Listener at localhost/44241] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 01:05:24,159 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 01:05:24,159 INFO [Listener at localhost/44241] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 01:05:24,159 INFO [Listener at localhost/44241] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 01:05:24,161 INFO [Listener at localhost/44241] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35243 2023-07-19 01:05:24,162 INFO [Listener at localhost/44241] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 01:05:24,163 DEBUG [Listener at localhost/44241] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 01:05:24,164 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:24,165 INFO [Listener at localhost/44241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 01:05:24,166 INFO [Listener at localhost/44241] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35243 connecting to ZooKeeper ensemble=127.0.0.1:53242 2023-07-19 01:05:24,169 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:352430x0, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 01:05:24,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:35243-0x1017b1c94f50008 connected 2023-07-19 01:05:24,170 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(162): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 01:05:24,171 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(162): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-19 01:05:24,172 DEBUG [Listener at localhost/44241] zookeeper.ZKUtil(164): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 01:05:24,173 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35243 2023-07-19 01:05:24,173 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35243 2023-07-19 01:05:24,173 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35243 2023-07-19 01:05:24,173 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35243 2023-07-19 01:05:24,174 DEBUG [Listener at localhost/44241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35243 2023-07-19 01:05:24,176 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(951): ClusterId : f10690fa-cb6d-420e-abdc-d71279a7071e 2023-07-19 01:05:24,177 DEBUG [RS:2;jenkins-hbase4:35243] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 01:05:24,179 DEBUG [RS:2;jenkins-hbase4:35243] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 01:05:24,179 DEBUG [RS:2;jenkins-hbase4:35243] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 01:05:24,181 DEBUG [RS:2;jenkins-hbase4:35243] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 01:05:24,182 DEBUG [RS:2;jenkins-hbase4:35243] zookeeper.ReadOnlyZKClient(139): Connect 0x0d7f48b2 to 127.0.0.1:53242 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 01:05:24,186 DEBUG [RS:2;jenkins-hbase4:35243] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b8a76d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 01:05:24,186 DEBUG [RS:2;jenkins-hbase4:35243] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78723c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 01:05:24,195 DEBUG [RS:2;jenkins-hbase4:35243] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:35243 2023-07-19 01:05:24,195 INFO [RS:2;jenkins-hbase4:35243] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 01:05:24,195 INFO [RS:2;jenkins-hbase4:35243] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 01:05:24,195 DEBUG [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 01:05:24,196 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,39087,1689728718150 with isa=jenkins-hbase4.apache.org/172.31.14.131:35243, startcode=1689728724158 2023-07-19 01:05:24,196 DEBUG [RS:2;jenkins-hbase4:35243] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 01:05:24,199 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40383, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 01:05:24,199 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39087] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:24,200 DEBUG [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081 2023-07-19 01:05:24,200 DEBUG [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:32877 2023-07-19 01:05:24,200 DEBUG [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 01:05:24,202 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:24,202 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:24,202 DEBUG [RS:2;jenkins-hbase4:35243] zookeeper.ZKUtil(162): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:24,202 WARN [RS:2;jenkins-hbase4:35243] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 01:05:24,202 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35243,1689728724158] 2023-07-19 01:05:24,202 INFO [RS:2;jenkins-hbase4:35243] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 01:05:24,203 DEBUG [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1948): logDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:24,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:24,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:24,208 DEBUG [RS:2;jenkins-hbase4:35243] zookeeper.ZKUtil(162): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:24,208 DEBUG [RS:2;jenkins-hbase4:35243] zookeeper.ZKUtil(162): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:24,209 DEBUG [RS:2;jenkins-hbase4:35243] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 01:05:24,209 INFO [RS:2;jenkins-hbase4:35243] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 01:05:24,219 INFO [RS:2;jenkins-hbase4:35243] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 01:05:24,219 INFO [RS:2;jenkins-hbase4:35243] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 01:05:24,219 INFO [RS:2;jenkins-hbase4:35243] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:24,220 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 01:05:24,221 INFO [RS:2;jenkins-hbase4:35243] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:24,222 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:24,222 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:24,222 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:24,222 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:24,222 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:24,223 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 01:05:24,223 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:24,223 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:24,223 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:24,223 DEBUG [RS:2;jenkins-hbase4:35243] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 01:05:24,224 INFO [RS:2;jenkins-hbase4:35243] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:24,224 INFO [RS:2;jenkins-hbase4:35243] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:24,224 INFO [RS:2;jenkins-hbase4:35243] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:24,236 INFO [RS:2;jenkins-hbase4:35243] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 01:05:24,236 INFO [RS:2;jenkins-hbase4:35243] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35243,1689728724158-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 01:05:24,248 INFO [RS:2;jenkins-hbase4:35243] regionserver.Replication(203): jenkins-hbase4.apache.org,35243,1689728724158 started 2023-07-19 01:05:24,248 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,35243,1689728724158, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35243, sessionid=0x1017b1c94f50008 2023-07-19 01:05:24,248 INFO [Listener at localhost/44241] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:35243,5,FailOnTimeoutGroup] 2023-07-19 01:05:24,248 DEBUG [RS:2;jenkins-hbase4:35243] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 01:05:24,248 DEBUG [RS:2;jenkins-hbase4:35243] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:24,248 INFO [Listener at localhost/44241] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:24,248 DEBUG [RS:2;jenkins-hbase4:35243] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35243,1689728724158' 2023-07-19 01:05:24,249 DEBUG [RS:2;jenkins-hbase4:35243] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 01:05:24,249 INFO [Listener at localhost/44241] client.TestScannerTimeout(177): START ************ test3686b 2023-07-19 01:05:24,249 DEBUG [RS:2;jenkins-hbase4:35243] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 01:05:24,250 DEBUG [RS:2;jenkins-hbase4:35243] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 01:05:24,250 DEBUG [RS:2;jenkins-hbase4:35243] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 01:05:24,250 DEBUG [RS:2;jenkins-hbase4:35243] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:24,250 DEBUG [RS:2;jenkins-hbase4:35243] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35243,1689728724158' 2023-07-19 01:05:24,250 DEBUG [RS:2;jenkins-hbase4:35243] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 01:05:24,251 DEBUG [RS:2;jenkins-hbase4:35243] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 01:05:24,251 DEBUG [RS:2;jenkins-hbase4:35243] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 01:05:24,251 INFO [RS:2;jenkins-hbase4:35243] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 01:05:24,251 INFO [RS:2;jenkins-hbase4:35243] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 01:05:24,252 DEBUG [Listener at localhost/44241] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 01:05:24,252 DEBUG [Listener at localhost/44241] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:24,253 DEBUG [Listener at localhost/44241] zookeeper.ReadOnlyZKClient(139): Connect 0x78946a4f to 127.0.0.1:53242 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 01:05:24,257 DEBUG [Listener at localhost/44241] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b6d62b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 01:05:24,260 DEBUG [hconnection-0x6ef1a5f0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:24,263 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51960, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:24,354 INFO [RS:2;jenkins-hbase4:35243] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35243%2C1689728724158, suffix=, logDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,35243,1689728724158, archiveDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs, maxLogs=32 2023-07-19 01:05:24,373 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK] 2023-07-19 01:05:24,373 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK] 2023-07-19 01:05:24,378 INFO [RS:2;jenkins-hbase4:35243] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,35243,1689728724158/jenkins-hbase4.apache.org%2C35243%2C1689728724158.1689728724355 2023-07-19 01:05:24,378 DEBUG [RS:2;jenkins-hbase4:35243] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK], DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK]] 2023-07-19 01:05:26,612 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-19 01:05:26,663 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-19 01:05:28,307 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 01:05:28,308 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-19 01:05:39,270 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 8595189265443323922 expired clientIPAndPort=172.31.14.131:51960, userName=jenkins, regionInfo=t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:41,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45105] regionserver.RSRpcServices(3094): Client tried to access missing scanner 8595189265443323922 2023-07-19 01:05:41,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45105] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:51960 deadline: 1689728801271, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8595189265443323922'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-19 01:05:41,273 DEBUG [Listener at localhost/44241] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8595189265443323922'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8595189265443323922'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8595189265443323922'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-19 01:05:41,278 DEBUG [Listener at localhost/44241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78946a4f to 127.0.0.1:53242 2023-07-19 01:05:41,278 DEBUG [Listener at localhost/44241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:41,278 INFO [Listener at localhost/44241] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-19 01:05:41,295 INFO [Listener at localhost/44241] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=296 (was 272) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53242@0x78946a4f-SendThread(127.0.0.1:53242) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: PacketResponder: BP-1782885210-172.31.14.131-1689728715060:blk_1073741843_1019, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1215921485_17 at /127.0.0.1:49846 [Receiving block BP-1782885210-172.31.14.131-1689728715060:blk_1073741843_1019] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081-prefix:jenkins-hbase4.apache.org,35243,1689728724158 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53242@0x78946a4f-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53242@0x0d7f48b2-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53242@0x0d7f48b2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/588131585.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1782885210-172.31.14.131-1689728715060:blk_1073741843_1019, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:32877 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase4:35243Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53242@0x78946a4f java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/588131585.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS:2;jenkins-hbase4:35243-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53242@0x0d7f48b2-SendThread(127.0.0.1:53242) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35243 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1215921485_17 at /127.0.0.1:34616 [Receiving block BP-1782885210-172.31.14.131-1689728715060:blk_1073741843_1019] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:35243 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=572 (was 575), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=133 (was 153), ProcessCount=171 (was 171), AvailableMemoryMB=6059 (was 5879) - AvailableMemoryMB LEAK? - 2023-07-19 01:05:41,316 INFO [Listener at localhost/44241] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=296, OpenFileDescriptor=572, MaxFileDescriptor=60000, SystemLoadAverage=133, ProcessCount=171, AvailableMemoryMB=6056 2023-07-19 01:05:41,317 INFO [Listener at localhost/44241] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:41065 2023-07-19 01:05:41,317 INFO [Listener at localhost/44241] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,41065,1689728719362 2023-07-19 01:05:41,317 INFO [Listener at localhost/44241] client.TestScannerTimeout(104): START************ test2772 2023-07-19 01:05:41,320 DEBUG [Listener at localhost/44241] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 01:05:41,320 DEBUG [Listener at localhost/44241] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:41,321 DEBUG [Listener at localhost/44241] zookeeper.ReadOnlyZKClient(139): Connect 0x5b2323dd to 127.0.0.1:53242 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 01:05:41,326 DEBUG [Listener at localhost/44241] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35c6779, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 01:05:41,326 ERROR [Listener at localhost/44241] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,45105,1689728719316: die! ***** 2023-07-19 01:05:41,327 ERROR [Listener at localhost/44241] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-19 01:05:41,327 DEBUG [Listener at localhost/44241] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-19 01:05:41,328 DEBUG [Listener at localhost/44241] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-19 01:05:41,333 DEBUG [Listener at localhost/44241] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-19 01:05:41,336 DEBUG [Listener at localhost/44241] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-19 01:05:41,355 INFO [Listener at localhost/44241] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 968884224, "init": 513802240, "max": 2051014656, "used": 279379616 }, "NonHeapMemoryUsage": { "committed": 122511360, "init": 2555904, "max": -1, "used": 120155464 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13024, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 63, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 63, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 63, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 63, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 63, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16683 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:53242", "tag.serverName": "jenkins-hbase4.apache.org,45105,1689728719316", "tag.clusterId": "f10690fa-cb6d-420e-abdc-d71279a7071e", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 2, "maxStoreFileCount": 1, "memStoreSize": 4186, "storeFileSize": 9852, "maxStoreFileAge": 16709, "minStoreFileAge": 16709, "avgStoreFileAge": 16709, "numReferenceFiles": 0, "regionServerStartTime": 1689728719316, "averageRegionSize": 4679, "storeFileIndexSize": 640, "staticIndexSize": 68, "staticBloomSize": 4, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1251, "localBytesRead": 1251, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 3314, "ByteBuffAllocatorPoolAllocationBytes": 1530880, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 41, "totalRowActionRequestCount": 51, "readRequestCount": 37, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 6, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 21, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 12, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 348, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 20, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 9852, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 20, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 348, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.607718472035003, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.2555682091660654, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-19 01:05:41,357 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39087] master.MasterRpcServices(612): jenkins-hbase4.apache.org,45105,1689728719316 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,45105,1689728719316: die! ***** 2023-07-19 01:05:41,358 INFO [Listener at localhost/44241] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,45105,1689728719316' ***** 2023-07-19 01:05:41,359 INFO [Listener at localhost/44241] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-19 01:05:41,359 INFO [RS:0;jenkins-hbase4:45105] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 01:05:41,359 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 01:05:41,359 INFO [RS:0;jenkins-hbase4:45105] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-19 01:05:41,360 INFO [RS:0;jenkins-hbase4:45105] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-19 01:05:41,360 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(3305): Received CLOSE for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:41,360 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(3305): Received CLOSE for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:41,360 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:41,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d8f2f6dea51dd237660eba5bde311772, disabling compactions & flushes 2023-07-19 01:05:41,360 DEBUG [RS:0;jenkins-hbase4:45105] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48048430 to 127.0.0.1:53242 2023-07-19 01:05:41,361 DEBUG [RS:0;jenkins-hbase4:45105] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:41,361 INFO [RS:0;jenkins-hbase4:45105] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 01:05:41,361 INFO [RS:0;jenkins-hbase4:45105] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 01:05:41,361 INFO [RS:0;jenkins-hbase4:45105] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 01:05:41,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:41,361 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-19 01:05:41,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:41,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. after waiting 0 ms 2023-07-19 01:05:41,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:41,362 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-19 01:05:41,362 DEBUG [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, d8f2f6dea51dd237660eba5bde311772=hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772., 9a00fc7c9da6b8edd769d227741a103a=t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.} 2023-07-19 01:05:41,362 DEBUG [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1504): Waiting on 1588230740, 9a00fc7c9da6b8edd769d227741a103a, d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:41,367 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 01:05:41,367 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 01:05:41,367 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 01:05:41,367 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 01:05:41,367 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 01:05:41,372 DEBUG [hconnection-0x55832593-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:41,372 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 4186 in region hbase:meta,,1.1588230740 2023-07-19 01:05:41,372 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 01:05:41,376 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 01:05:41,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:41,376 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d8f2f6dea51dd237660eba5bde311772: 2023-07-19 01:05:41,376 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 01:05:41,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:41,377 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-19 01:05:41,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9a00fc7c9da6b8edd769d227741a103a, disabling compactions & flushes 2023-07-19 01:05:41,377 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:41,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:41,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. after waiting 0 ms 2023-07-19 01:05:41,377 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48610, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:41,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:41,378 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45105] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45105,1689728719316 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 01:05:41,378 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45105] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:48610 deadline: 1689728801378, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45105,1689728719316 aborting 2023-07-19 01:05:41,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:41,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9a00fc7c9da6b8edd769d227741a103a: 2023-07-19 01:05:41,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:41,486 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45105] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45105,1689728719316 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 01:05:41,486 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45105] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:48610 deadline: 1689728801485, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45105,1689728719316 aborting 2023-07-19 01:05:41,562 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,45105,1689728719316; all regions closed. 2023-07-19 01:05:41,585 DEBUG [RS:0;jenkins-hbase4:45105] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:41,586 INFO [RS:0;jenkins-hbase4:45105] regionserver.LeaseManager(133): Closed leases 2023-07-19 01:05:41,586 INFO [RS:0;jenkins-hbase4:45105] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-19 01:05:41,586 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 01:05:41,587 INFO [RS:0;jenkins-hbase4:45105] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45105 2023-07-19 01:05:41,591 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:41,591 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:41,591 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 2023-07-19 01:05:41,592 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:41,592 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:41,593 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,45105,1689728719316] 2023-07-19 01:05:41,593 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,45105,1689728719316; numProcessing=1 2023-07-19 01:05:41,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:41,594 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,45105,1689728719316 znode expired, triggering replicatorRemoved event 2023-07-19 01:05:41,595 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,45105,1689728719316 already deleted, retry=false 2023-07-19 01:05:41,595 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,45105,1689728719316 on jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:41,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:41,596 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,45105,1689728719316, splitWal=true, meta=true 2023-07-19 01:05:41,597 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=19 for jenkins-hbase4.apache.org,45105,1689728719316 (carryingMeta=true) jenkins-hbase4.apache.org,45105,1689728719316/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2c79eadf[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-19 01:05:41,599 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=19, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45105,1689728719316, splitWal=true, meta=true 2023-07-19 01:05:41,601 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45105,1689728719316, splitWal=true, meta=true, isMeta: true 2023-07-19 01:05:41,608 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting 2023-07-19 01:05:41,611 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,45105,1689728719316 WAL count=1, meta=true 2023-07-19 01:05:41,611 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta}] 2023-07-19 01:05:41,615 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:41,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta, worker=jenkins-hbase4.apache.org,35243,1689728724158}] 2023-07-19 01:05:41,689 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:45105 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45105 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 01:05:41,690 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 01:05:41,690 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:45105 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45105 2023-07-19 01:05:41,693 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 01:05:41,693 INFO [RS:0;jenkins-hbase4:45105] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,45105,1689728719316; zookeeper connection closed. 2023-07-19 01:05:41,693 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:45105-0x1017b1c94f50001, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 01:05:41,694 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@49c5adc] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@49c5adc 2023-07-19 01:05:41,771 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:41,771 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 01:05:41,773 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 01:05:41,774 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35243] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=21 2023-07-19 01:05:41,796 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta, size=5.1 K (5185bytes) 2023-07-19 01:05:41,796 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta 2023-07-19 01:05:41,797 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta after 1ms 2023-07-19 01:05:41,807 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta took 11ms 2023-07-19 01:05:41,823 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-19 01:05:41,825 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta so closing down 2023-07-19 01:05:41,826 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 01:05:41,830 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta.temp 2023-07-19 01:05:41,831 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 01:05:41,849 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta.temp (wrote 14 edits, skipped 0 edits in 0 ms) 2023-07-19 01:05:41,851 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta.temp to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-19 01:05:41,851 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 15 edits across 1 Regions in 33 ms; skipped=1; WAL=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta, size=5.1 K, length=5185, corrupted=false, cancelled=false 2023-07-19 01:05:41,851 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta, journal: Splitting hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta, size=5.1 K (5185bytes) at 1689728741796Finishing writing output for hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta so closing down at 1689728741826 (+30 ms)Creating recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta.temp at 1689728741830 (+4 ms)3 split writer threads finished at 1689728741831 (+1 ms)Closed recovered edits writer path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta.temp (wrote 14 edits, skipped 0 edits in 0 ms) at 1689728741849 (+18 ms)Rename recovered edits hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta.temp to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 at 1689728741851 (+2 ms)Processed 15 edits across 1 Regions in 33 ms; skipped=1; WAL=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta, size=5.1 K, length=5185, corrupted=false, cancelled=false at 1689728741851 2023-07-19 01:05:41,852 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta 2023-07-19 01:05:41,852 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=21 2023-07-19 01:05:41,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] master.HMaster(3641): Remote procedure done, pid=21 2023-07-19 01:05:41,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] wal.WALSplitUtil(141): Moved hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs 2023-07-19 01:05:41,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-19 01:05:41,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta, worker=jenkins-hbase4.apache.org,35243,1689728724158 in 241 msec 2023-07-19 01:05:41,864 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:41,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-19 01:05:41,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C45105%2C1689728719316.meta.1689728720992.meta, worker=jenkins-hbase4.apache.org,35243,1689728724158 in 253 msec 2023-07-19 01:05:41,868 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting dir is empty, no logs to split. 2023-07-19 01:05:41,868 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,45105,1689728719316 WAL count=0, meta=true 2023-07-19 01:05:41,869 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,45105,1689728719316 WAL splitting is done? wals=0, meta=true 2023-07-19 01:05:41,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-19 01:05:41,871 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-19 01:05:41,872 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 01:05:41,994 DEBUG [hconnection-0x55832593-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45105 this server is in the failed servers list 2023-07-19 01:05:42,022 DEBUG [jenkins-hbase4:39087] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 01:05:42,022 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35243,1689728724158, state=OPENING 2023-07-19 01:05:42,024 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 01:05:42,024 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 01:05:42,024 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35243,1689728724158}] 2023-07-19 01:05:42,185 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-19 01:05:42,185 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 01:05:42,187 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35243%2C1689728724158.meta, suffix=.meta, logDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,35243,1689728724158, archiveDir=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs, maxLogs=32 2023-07-19 01:05:42,214 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK] 2023-07-19 01:05:42,216 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK] 2023-07-19 01:05:42,219 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,35243,1689728724158/jenkins-hbase4.apache.org%2C35243%2C1689728724158.meta.1689728742189.meta 2023-07-19 01:05:42,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38291,DS-78cfb6f1-5dbd-4fb0-b5f0-6c9ce2f9eb93,DISK], DatanodeInfoWithStorage[127.0.0.1:45855,DS-3a4fc61c-e003-48df-9f55-2cc5b85a5228,DISK]] 2023-07-19 01:05:42,223 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:42,223 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-19 01:05:42,223 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-19 01:05:42,224 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-19 01:05:42,224 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-19 01:05:42,224 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:42,224 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-19 01:05:42,224 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-19 01:05:42,231 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 01:05:42,234 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/info 2023-07-19 01:05:42,234 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/info 2023-07-19 01:05:42,234 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 01:05:42,235 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:42,235 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 01:05:42,239 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/rep_barrier 2023-07-19 01:05:42,239 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/rep_barrier 2023-07-19 01:05:42,239 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 01:05:42,241 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:42,241 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 01:05:42,246 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/table 2023-07-19 01:05:42,246 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/table 2023-07-19 01:05:42,247 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 01:05:42,248 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:42,249 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740 2023-07-19 01:05:42,252 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740 2023-07-19 01:05:42,253 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-19 01:05:42,260 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 40, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=17, path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-19 01:05:42,261 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=4.09 KB heapSize=7.79 KB 2023-07-19 01:05:42,291 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.93 KB at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/.tmp/info/0f7e87eee07b4d2b994a7eac561b1f0a 2023-07-19 01:05:42,377 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/.tmp/table/2f1cd63d7833440b8640fbc3264f8cd0 2023-07-19 01:05:42,389 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/.tmp/info/0f7e87eee07b4d2b994a7eac561b1f0a as hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/info/0f7e87eee07b4d2b994a7eac561b1f0a 2023-07-19 01:05:42,401 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/info/0f7e87eee07b4d2b994a7eac561b1f0a, entries=36, sequenceid=17, filesize=8.6 K 2023-07-19 01:05:42,403 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/.tmp/table/2f1cd63d7833440b8640fbc3264f8cd0 as hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/table/2f1cd63d7833440b8640fbc3264f8cd0 2023-07-19 01:05:42,412 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/table/2f1cd63d7833440b8640fbc3264f8cd0, entries=4, sequenceid=17, filesize=4.7 K 2023-07-19 01:05:42,412 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~4.09 KB/4186, heapSize ~7.51 KB/7688, currentSize=0 B/0 for 1588230740 in 151ms, sequenceid=17, compaction requested=false; wal=null 2023-07-19 01:05:42,414 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-19 01:05:42,417 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 01:05:42,419 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 01:05:42,422 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=1 2023-07-19 01:05:42,424 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=18; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68084822, jitterRate=0.014542907476425171}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 01:05:42,424 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 01:05:42,425 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=23, masterSystemTime=1689728742177 2023-07-19 01:05:42,428 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-19 01:05:42,428 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-19 01:05:42,429 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35243,1689728724158, state=OPEN 2023-07-19 01:05:42,430 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 01:05:42,430 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 01:05:42,433 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-19 01:05:42,433 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35243,1689728724158 in 406 msec 2023-07-19 01:05:42,436 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-07-19 01:05:42,436 INFO [PEWorker-1] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,45105,1689728719316 had 3 regions 2023-07-19 01:05:42,436 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 563 msec 2023-07-19 01:05:42,437 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45105,1689728719316, splitWal=true, meta=true, isMeta: false 2023-07-19 01:05:42,439 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,45105,1689728719316 WAL count=1, meta=false 2023-07-19 01:05:42,439 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=19, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824}] 2023-07-19 01:05:42,441 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:42,442 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824, worker=jenkins-hbase4.apache.org,35243,1689728724158}] 2023-07-19 01:05:42,499 DEBUG [hconnection-0x55832593-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:42,502 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:42,509 DEBUG [hconnection-0x55832593-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45105 this server is in the failed servers list 2023-07-19 01:05:42,595 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=35243] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-19 01:05:42,608 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824, size=721 (721bytes) 2023-07-19 01:05:42,609 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824 2023-07-19 01:05:42,609 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824 after 0ms 2023-07-19 01:05:42,615 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824 took 7ms 2023-07-19 01:05:42,619 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 9a00fc7c9da6b8edd769d227741a103a: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-19 01:05:42,621 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for d8f2f6dea51dd237660eba5bde311772: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-19 01:05:42,621 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824 so closing down 2023-07-19 01:05:42,621 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 01:05:42,621 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 01:05:42,622 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 6 ms; skipped=2; WAL=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824, size=721, length=721, corrupted=false, cancelled=false 2023-07-19 01:05:42,622 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824, journal: Splitting hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824, size=721 (721bytes) at 1689728742609Finishing writing output for hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824 so closing down at 1689728742621 (+12 ms)3 split writer threads finished at 1689728742622 (+1 ms)Processed 2 edits across 0 Regions in 6 ms; skipped=2; WAL=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824, size=721, length=721, corrupted=false, cancelled=false at 1689728742622 2023-07-19 01:05:42,622 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824 2023-07-19 01:05:42,622 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-19 01:05:42,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] master.HMaster(3641): Remote procedure done, pid=25 2023-07-19 01:05:42,623 DEBUG [hconnection-0x55832593-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45105 this server is in the failed servers list 2023-07-19 01:05:42,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39087] wal.WALSplitUtil(141): Moved hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting/jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824 to hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs 2023-07-19 01:05:42,629 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-19 01:05:42,629 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824, worker=jenkins-hbase4.apache.org,35243,1689728724158 in 185 msec 2023-07-19 01:05:42,631 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:42,633 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=19 2023-07-19 01:05:42,634 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=19, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C45105%2C1689728719316.1689728720824, worker=jenkins-hbase4.apache.org,35243,1689728724158 in 192 msec 2023-07-19 01:05:42,636 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/WALs/jenkins-hbase4.apache.org,45105,1689728719316-splitting dir is empty, no logs to split. 2023-07-19 01:05:42,636 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,45105,1689728719316 WAL count=0, meta=false 2023-07-19 01:05:42,636 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,45105,1689728719316 WAL splitting is done? wals=0, meta=false 2023-07-19 01:05:42,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN}, {pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN}] 2023-07-19 01:05:42,639 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN 2023-07-19 01:05:42,639 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN 2023-07-19 01:05:42,640 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 01:05:42,641 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 01:05:42,791 DEBUG [jenkins-hbase4:39087] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 01:05:42,791 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=9a00fc7c9da6b8edd769d227741a103a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:42,791 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=d8f2f6dea51dd237660eba5bde311772, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:42,792 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689728742791"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689728742791"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689728742791"}]},"ts":"1689728742791"} 2023-07-19 01:05:42,792 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689728742791"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689728742791"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689728742791"}]},"ts":"1689728742791"} 2023-07-19 01:05:42,793 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:45105 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45105 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 01:05:42,794 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:45105 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45105 2023-07-19 01:05:42,835 DEBUG [hconnection-0x55832593-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45105 this server is in the failed servers list 2023-07-19 01:05:42,897 DEBUG [PEWorker-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 01:05:42,899 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 01:05:42,902 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE; OpenRegionProcedure d8f2f6dea51dd237660eba5bde311772, server=jenkins-hbase4.apache.org,35243,1689728724158}] 2023-07-19 01:05:42,904 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=26, state=RUNNABLE; OpenRegionProcedure 9a00fc7c9da6b8edd769d227741a103a, server=jenkins-hbase4.apache.org,35243,1689728724158}] 2023-07-19 01:05:43,069 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:43,069 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d8f2f6dea51dd237660eba5bde311772, NAME => 'hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:43,070 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:43,070 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:43,070 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:43,070 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:43,072 INFO [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:43,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:43,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9a00fc7c9da6b8edd769d227741a103a, NAME => 't,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.', STARTKEY => '', ENDKEY => ''} 2023-07-19 01:05:43,074 DEBUG [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/info 2023-07-19 01:05:43,074 DEBUG [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/info 2023-07-19 01:05:43,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:43,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 01:05:43,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:43,075 INFO [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d8f2f6dea51dd237660eba5bde311772 columnFamilyName info 2023-07-19 01:05:43,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:43,076 INFO [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:43,080 DEBUG [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/f 2023-07-19 01:05:43,080 DEBUG [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/f 2023-07-19 01:05:43,081 INFO [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a00fc7c9da6b8edd769d227741a103a columnFamilyName f 2023-07-19 01:05:43,092 DEBUG [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] regionserver.HStore(539): loaded hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/info/11e7afd179ba466e9876ce45532063f2 2023-07-19 01:05:43,092 INFO [StoreOpener-d8f2f6dea51dd237660eba5bde311772-1] regionserver.HStore(310): Store=d8f2f6dea51dd237660eba5bde311772/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:43,094 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:43,096 DEBUG [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] regionserver.HStore(539): loaded hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/f/d98835a4e6e24223b5b213d5024b152b 2023-07-19 01:05:43,096 INFO [StoreOpener-9a00fc7c9da6b8edd769d227741a103a-1] regionserver.HStore(310): Store=9a00fc7c9da6b8edd769d227741a103a/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 01:05:43,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:43,100 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:43,101 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:43,105 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:43,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:43,109 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-19 01:05:43,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-19 01:05:43,110 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d8f2f6dea51dd237660eba5bde311772; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61651089, jitterRate=-0.08132718503475189}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 01:05:43,110 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d8f2f6dea51dd237660eba5bde311772: 2023-07-19 01:05:43,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9a00fc7c9da6b8edd769d227741a103a; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60040884, jitterRate=-0.10532110929489136}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 01:05:43,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9a00fc7c9da6b8edd769d227741a103a: 2023-07-19 01:05:43,111 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772., pid=28, masterSystemTime=1689728743063 2023-07-19 01:05:43,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a., pid=29, masterSystemTime=1689728743063 2023-07-19 01:05:43,114 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:43,114 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:43,115 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=d8f2f6dea51dd237660eba5bde311772, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:43,115 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689728743114"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689728743114"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689728743114"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689728743114"}]},"ts":"1689728743114"} 2023-07-19 01:05:43,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:43,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:43,116 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=9a00fc7c9da6b8edd769d227741a103a, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:43,116 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689728743115"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689728743115"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689728743115"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689728743115"}]},"ts":"1689728743115"} 2023-07-19 01:05:43,120 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=27 2023-07-19 01:05:43,120 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; OpenRegionProcedure d8f2f6dea51dd237660eba5bde311772, server=jenkins-hbase4.apache.org,35243,1689728724158 in 216 msec 2023-07-19 01:05:43,125 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=26 2023-07-19 01:05:43,125 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=26, state=SUCCESS; OpenRegionProcedure 9a00fc7c9da6b8edd769d227741a103a, server=jenkins-hbase4.apache.org,35243,1689728724158 in 215 msec 2023-07-19 01:05:43,125 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d8f2f6dea51dd237660eba5bde311772, ASSIGN in 482 msec 2023-07-19 01:05:43,127 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-19 01:05:43,128 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,45105,1689728719316 after splitting done 2023-07-19 01:05:43,128 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=t, region=9a00fc7c9da6b8edd769d227741a103a, ASSIGN in 487 msec 2023-07-19 01:05:43,128 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase4.apache.org,45105,1689728719316 from processing; numProcessing=0 2023-07-19 01:05:43,130 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,45105,1689728719316, splitWal=true, meta=true in 1.5330 sec 2023-07-19 01:05:43,147 DEBUG [Listener at localhost/44241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b2323dd to 127.0.0.1:53242 2023-07-19 01:05:43,147 DEBUG [Listener at localhost/44241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:43,147 INFO [Listener at localhost/44241] client.TestScannerTimeout(124): END ************ test2772 2023-07-19 01:05:43,162 INFO [Listener at localhost/44241] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 296), OpenFileDescriptor=577 (was 572) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=133 (was 133), ProcessCount=171 (was 171), AvailableMemoryMB=6051 (was 6056) 2023-07-19 01:05:43,163 INFO [Listener at localhost/44241] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-19 01:05:43,163 INFO [Listener at localhost/44241] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-19 01:05:43,164 DEBUG [Listener at localhost/44241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f87e9f1 to 127.0.0.1:53242 2023-07-19 01:05:43,164 DEBUG [Listener at localhost/44241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:43,164 DEBUG [Listener at localhost/44241] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-19 01:05:43,164 DEBUG [Listener at localhost/44241] util.JVMClusterUtil(257): Found active master hash=1367985792, stopped=false 2023-07-19 01:05:43,164 INFO [Listener at localhost/44241] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:43,171 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-19 01:05:43,171 INFO [Listener at localhost/44241] procedure2.ProcedureExecutor(629): Stopping 2023-07-19 01:05:43,171 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-19 01:05:43,171 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:43,171 DEBUG [Listener at localhost/44241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1cc6ac18 to 127.0.0.1:53242 2023-07-19 01:05:43,171 DEBUG [Listener at localhost/44241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:43,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 01:05:43,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 01:05:43,172 INFO [Listener at localhost/44241] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,35243,1689728724158' ***** 2023-07-19 01:05:43,172 INFO [Listener at localhost/44241] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-19 01:05:43,172 INFO [RS:2;jenkins-hbase4:35243] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 01:05:43,172 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 01:05:43,172 INFO [RS:2;jenkins-hbase4:35243] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-19 01:05:43,172 INFO [RS:2;jenkins-hbase4:35243] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-19 01:05:43,172 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(3305): Received CLOSE for d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:43,173 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(3305): Received CLOSE for 9a00fc7c9da6b8edd769d227741a103a 2023-07-19 01:05:43,173 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:43,173 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d8f2f6dea51dd237660eba5bde311772, disabling compactions & flushes 2023-07-19 01:05:43,173 DEBUG [RS:2;jenkins-hbase4:35243] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d7f48b2 to 127.0.0.1:53242 2023-07-19 01:05:43,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:43,173 DEBUG [RS:2;jenkins-hbase4:35243] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:43,173 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:43,173 INFO [RS:2;jenkins-hbase4:35243] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 01:05:43,173 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. after waiting 0 ms 2023-07-19 01:05:43,174 INFO [RS:2;jenkins-hbase4:35243] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 01:05:43,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:43,174 INFO [RS:2;jenkins-hbase4:35243] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 01:05:43,174 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-19 01:05:43,174 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-19 01:05:43,174 DEBUG [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, d8f2f6dea51dd237660eba5bde311772=hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772., 9a00fc7c9da6b8edd769d227741a103a=t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a.} 2023-07-19 01:05:43,174 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 01:05:43,174 DEBUG [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1504): Waiting on 1588230740, 9a00fc7c9da6b8edd769d227741a103a, d8f2f6dea51dd237660eba5bde311772 2023-07-19 01:05:43,175 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 01:05:43,175 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 01:05:43,175 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 01:05:43,176 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 01:05:43,176 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-19 01:05:43,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/namespace/d8f2f6dea51dd237660eba5bde311772/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-19 01:05:43,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:43,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d8f2f6dea51dd237660eba5bde311772: 2023-07-19 01:05:43,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689728721303.d8f2f6dea51dd237660eba5bde311772. 2023-07-19 01:05:43,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9a00fc7c9da6b8edd769d227741a103a, disabling compactions & flushes 2023-07-19 01:05:43,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:43,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:43,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. after waiting 0 ms 2023-07-19 01:05:43,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:43,188 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/default/t/9a00fc7c9da6b8edd769d227741a103a/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-19 01:05:43,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:43,189 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9a00fc7c9da6b8edd769d227741a103a: 2023-07-19 01:05:43,190 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689728722191.9a00fc7c9da6b8edd769d227741a103a. 2023-07-19 01:05:43,203 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/.tmp/info/1c661fdf975f4a96a68d10003fe42155 2023-07-19 01:05:43,211 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/.tmp/info/1c661fdf975f4a96a68d10003fe42155 as hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/info/1c661fdf975f4a96a68d10003fe42155 2023-07-19 01:05:43,218 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/info/1c661fdf975f4a96a68d10003fe42155, entries=16, sequenceid=24, filesize=6.4 K 2023-07-19 01:05:43,219 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 43ms, sequenceid=24, compaction requested=false 2023-07-19 01:05:43,224 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-19 01:05:43,225 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-19 01:05:43,226 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=17 2023-07-19 01:05:43,226 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 01:05:43,226 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 01:05:43,226 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 01:05:43,226 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 01:05:43,226 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-19 01:05:43,375 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,35243,1689728724158; all regions closed. 2023-07-19 01:05:43,382 DEBUG [RS:2;jenkins-hbase4:35243] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs 2023-07-19 01:05:43,382 INFO [RS:2;jenkins-hbase4:35243] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35243%2C1689728724158.meta:.meta(num 1689728742189) 2023-07-19 01:05:43,387 DEBUG [RS:2;jenkins-hbase4:35243] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/oldWALs 2023-07-19 01:05:43,387 INFO [RS:2;jenkins-hbase4:35243] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35243%2C1689728724158:(num 1689728724355) 2023-07-19 01:05:43,387 DEBUG [RS:2;jenkins-hbase4:35243] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:43,387 INFO [RS:2;jenkins-hbase4:35243] regionserver.LeaseManager(133): Closed leases 2023-07-19 01:05:43,388 INFO [RS:2;jenkins-hbase4:35243] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-19 01:05:43,388 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 01:05:43,389 INFO [RS:2;jenkins-hbase4:35243] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35243 2023-07-19 01:05:43,391 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:43,391 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35243,1689728724158 2023-07-19 01:05:43,391 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 01:05:43,393 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35243,1689728724158] 2023-07-19 01:05:43,393 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35243,1689728724158; numProcessing=1 2023-07-19 01:05:43,394 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35243,1689728724158 already deleted, retry=false 2023-07-19 01:05:43,394 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35243,1689728724158 expired; onlineServers=0 2023-07-19 01:05:43,394 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39087,1689728718150' ***** 2023-07-19 01:05:43,394 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-19 01:05:43,395 DEBUG [M:0;jenkins-hbase4:39087] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10b4c4bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 01:05:43,395 INFO [M:0;jenkins-hbase4:39087] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,39087,1689728718150 2023-07-19 01:05:43,395 INFO [M:0;jenkins-hbase4:39087] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,39087,1689728718150; all regions closed. 2023-07-19 01:05:43,395 DEBUG [M:0;jenkins-hbase4:39087] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 01:05:43,395 DEBUG [M:0;jenkins-hbase4:39087] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-19 01:05:43,396 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-19 01:05:43,396 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689728720483] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689728720483,5,FailOnTimeoutGroup] 2023-07-19 01:05:43,396 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689728720482] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689728720482,5,FailOnTimeoutGroup] 2023-07-19 01:05:43,396 DEBUG [M:0;jenkins-hbase4:39087] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-19 01:05:43,396 INFO [M:0;jenkins-hbase4:39087] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-19 01:05:43,396 INFO [M:0;jenkins-hbase4:39087] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-19 01:05:43,396 INFO [M:0;jenkins-hbase4:39087] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-19 01:05:43,396 DEBUG [M:0;jenkins-hbase4:39087] master.HMaster(1512): Stopping service threads 2023-07-19 01:05:43,396 INFO [M:0;jenkins-hbase4:39087] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-19 01:05:43,397 ERROR [M:0;jenkins-hbase4:39087] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (90527944) connection to localhost/127.0.0.1:32877 from jenkins,5,PEWorkerGroup] 2023-07-19 01:05:43,397 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-19 01:05:43,397 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 01:05:43,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 01:05:43,397 INFO [M:0;jenkins-hbase4:39087] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-19 01:05:43,397 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-19 01:05:43,398 DEBUG [M:0;jenkins-hbase4:39087] zookeeper.ZKUtil(398): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-19 01:05:43,398 WARN [M:0;jenkins-hbase4:39087] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-19 01:05:43,398 INFO [M:0;jenkins-hbase4:39087] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-19 01:05:43,398 INFO [M:0;jenkins-hbase4:39087] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-19 01:05:43,398 DEBUG [M:0;jenkins-hbase4:39087] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-19 01:05:43,399 INFO [M:0;jenkins-hbase4:39087] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 01:05:43,399 DEBUG [M:0;jenkins-hbase4:39087] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 01:05:43,399 DEBUG [M:0;jenkins-hbase4:39087] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-19 01:05:43,399 DEBUG [M:0;jenkins-hbase4:39087] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 01:05:43,399 INFO [M:0;jenkins-hbase4:39087] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=96.35 KB heapSize=116.35 KB 2023-07-19 01:05:43,416 INFO [M:0;jenkins-hbase4:39087] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=96.35 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d1ce90f06bb047ed80db78c59bfb01c4 2023-07-19 01:05:43,423 DEBUG [M:0;jenkins-hbase4:39087] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d1ce90f06bb047ed80db78c59bfb01c4 as hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d1ce90f06bb047ed80db78c59bfb01c4 2023-07-19 01:05:43,431 INFO [M:0;jenkins-hbase4:39087] regionserver.HStore(1080): Added hdfs://localhost:32877/user/jenkins/test-data/2539db63-1a6b-308d-3876-60ca90f03081/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d1ce90f06bb047ed80db78c59bfb01c4, entries=29, sequenceid=225, filesize=8.3 K 2023-07-19 01:05:43,432 INFO [M:0;jenkins-hbase4:39087] regionserver.HRegion(2948): Finished flush of dataSize ~96.35 KB/98666, heapSize ~116.34 KB/119128, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 33ms, sequenceid=225, compaction requested=false 2023-07-19 01:05:43,433 INFO [M:0;jenkins-hbase4:39087] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 01:05:43,433 DEBUG [M:0;jenkins-hbase4:39087] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 01:05:43,437 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 01:05:43,437 INFO [M:0;jenkins-hbase4:39087] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-19 01:05:43,438 INFO [M:0;jenkins-hbase4:39087] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39087 2023-07-19 01:05:43,440 DEBUG [M:0;jenkins-hbase4:39087] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39087,1689728718150 already deleted, retry=false 2023-07-19 01:05:43,651 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 01:05:43,651 INFO [M:0;jenkins-hbase4:39087] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39087,1689728718150; zookeeper connection closed. 2023-07-19 01:05:43,651 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): master:39087-0x1017b1c94f50000, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 01:05:43,752 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 01:05:43,752 INFO [RS:2;jenkins-hbase4:35243] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,35243,1689728724158; zookeeper connection closed. 2023-07-19 01:05:43,752 DEBUG [Listener at localhost/44241-EventThread] zookeeper.ZKWatcher(600): regionserver:35243-0x1017b1c94f50008, quorum=127.0.0.1:53242, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 01:05:43,752 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@30e6fe36] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@30e6fe36 2023-07-19 01:05:43,752 INFO [Listener at localhost/44241] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-19 01:05:43,753 WARN [Listener at localhost/44241] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-19 01:05:43,755 INFO [Listener at localhost/44241] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 01:05:43,860 WARN [BP-1782885210-172.31.14.131-1689728715060 heartbeating to localhost/127.0.0.1:32877] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-19 01:05:43,860 WARN [BP-1782885210-172.31.14.131-1689728715060 heartbeating to localhost/127.0.0.1:32877] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1782885210-172.31.14.131-1689728715060 (Datanode Uuid 57050ffe-2818-43d1-8aaa-eaa23fe4d1a5) service to localhost/127.0.0.1:32877 2023-07-19 01:05:43,862 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/dfs/data/data3/current/BP-1782885210-172.31.14.131-1689728715060] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 01:05:43,862 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/dfs/data/data4/current/BP-1782885210-172.31.14.131-1689728715060] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 01:05:43,864 WARN [Listener at localhost/44241] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-19 01:05:43,867 INFO [Listener at localhost/44241] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 01:05:43,973 WARN [BP-1782885210-172.31.14.131-1689728715060 heartbeating to localhost/127.0.0.1:32877] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-19 01:05:43,973 WARN [BP-1782885210-172.31.14.131-1689728715060 heartbeating to localhost/127.0.0.1:32877] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1782885210-172.31.14.131-1689728715060 (Datanode Uuid c9b5721b-1253-49ab-b925-2d13db55f627) service to localhost/127.0.0.1:32877 2023-07-19 01:05:43,974 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/dfs/data/data1/current/BP-1782885210-172.31.14.131-1689728715060] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 01:05:43,975 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1977a6f8-cf64-15cf-0729-29d44d4f0d0f/cluster_c9b14d30-7be8-14f5-f6b2-b97b2e3347d3/dfs/data/data2/current/BP-1782885210-172.31.14.131-1689728715060] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 01:05:44,018 INFO [Listener at localhost/44241] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 01:05:44,131 INFO [Listener at localhost/44241] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-19 01:05:44,164 INFO [Listener at localhost/44241] hbase.HBaseTestingUtility(1293): Minicluster is down