2023-07-14 07:06:37,693 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-14 07:06:37,938 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b 2023-07-14 07:06:37,951 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-14 07:06:37,951 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278, deleteOnExit=true 2023-07-14 07:06:37,952 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-14 07:06:37,952 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/test.cache.data in system properties and HBase conf 2023-07-14 07:06:37,952 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/hadoop.tmp.dir in system properties and HBase conf 2023-07-14 07:06:37,953 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/hadoop.log.dir in system properties and HBase conf 2023-07-14 07:06:37,953 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-14 07:06:37,954 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-14 07:06:37,954 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-14 07:06:38,077 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-14 07:06:38,535 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-14 07:06:38,540 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-14 07:06:38,540 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-14 07:06:38,541 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-14 07:06:38,541 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-14 07:06:38,541 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-14 07:06:38,541 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-14 07:06:38,542 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-14 07:06:38,542 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-14 07:06:38,542 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-14 07:06:38,543 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/nfs.dump.dir in system properties and HBase conf 2023-07-14 07:06:38,543 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/java.io.tmpdir in system properties and HBase conf 2023-07-14 07:06:38,543 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-14 07:06:38,543 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-14 07:06:38,544 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-14 07:06:39,016 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-14 07:06:39,020 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-14 07:06:39,301 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-14 07:06:39,481 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-14 07:06:39,495 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 07:06:39,528 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 07:06:39,560 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/java.io.tmpdir/Jetty_localhost_37455_hdfs____.751h0m/webapp 2023-07-14 07:06:39,695 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37455 2023-07-14 07:06:39,711 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-14 07:06:39,711 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-14 07:06:40,205 WARN [Listener at localhost/35789] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 07:06:40,268 WARN [Listener at localhost/35789] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-14 07:06:40,293 WARN [Listener at localhost/35789] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 07:06:40,301 INFO [Listener at localhost/35789] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 07:06:40,308 INFO [Listener at localhost/35789] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/java.io.tmpdir/Jetty_localhost_39989_datanode____sljyd7/webapp 2023-07-14 07:06:40,410 INFO [Listener at localhost/35789] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39989 2023-07-14 07:06:40,712 WARN [Listener at localhost/45461] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 07:06:40,723 WARN [Listener at localhost/45461] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-14 07:06:40,725 WARN [Listener at localhost/45461] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 07:06:40,728 INFO [Listener at localhost/45461] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 07:06:40,735 INFO [Listener at localhost/45461] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/java.io.tmpdir/Jetty_localhost_42577_datanode____sf4lxa/webapp 2023-07-14 07:06:40,836 INFO [Listener at localhost/45461] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42577 2023-07-14 07:06:40,847 WARN [Listener at localhost/46223] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 07:06:41,163 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xafa02f5718cbc7fe: Processing first storage report for DS-0c17373b-6185-47ab-b29d-cc71331fb632 from datanode 58e3e07d-dd68-4956-9014-3d9ed84cd780 2023-07-14 07:06:41,165 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xafa02f5718cbc7fe: from storage DS-0c17373b-6185-47ab-b29d-cc71331fb632 node DatanodeRegistration(127.0.0.1:33815, datanodeUuid=58e3e07d-dd68-4956-9014-3d9ed84cd780, infoPort=42015, infoSecurePort=0, ipcPort=46223, storageInfo=lv=-57;cid=testClusterID;nsid=313737442;c=1689318399097), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-14 07:06:41,165 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xdf67481a78a284a0: Processing first storage report for DS-7a838618-123f-4d6b-86d0-84d6291e9886 from datanode 0d61d807-3a42-401f-964f-0e3b86d434ea 2023-07-14 07:06:41,165 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xdf67481a78a284a0: from storage DS-7a838618-123f-4d6b-86d0-84d6291e9886 node DatanodeRegistration(127.0.0.1:41169, datanodeUuid=0d61d807-3a42-401f-964f-0e3b86d434ea, infoPort=45307, infoSecurePort=0, ipcPort=45461, storageInfo=lv=-57;cid=testClusterID;nsid=313737442;c=1689318399097), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 07:06:41,165 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xafa02f5718cbc7fe: Processing first storage report for DS-39727713-e143-49d4-b46c-16c8851c8770 from datanode 58e3e07d-dd68-4956-9014-3d9ed84cd780 2023-07-14 07:06:41,165 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xafa02f5718cbc7fe: from storage DS-39727713-e143-49d4-b46c-16c8851c8770 node DatanodeRegistration(127.0.0.1:33815, datanodeUuid=58e3e07d-dd68-4956-9014-3d9ed84cd780, infoPort=42015, infoSecurePort=0, ipcPort=46223, storageInfo=lv=-57;cid=testClusterID;nsid=313737442;c=1689318399097), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 07:06:41,166 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xdf67481a78a284a0: Processing first storage report for DS-1436a946-9e79-422a-9da3-d9d6ddf5dee9 from datanode 0d61d807-3a42-401f-964f-0e3b86d434ea 2023-07-14 07:06:41,166 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xdf67481a78a284a0: from storage DS-1436a946-9e79-422a-9da3-d9d6ddf5dee9 node DatanodeRegistration(127.0.0.1:41169, datanodeUuid=0d61d807-3a42-401f-964f-0e3b86d434ea, infoPort=45307, infoSecurePort=0, ipcPort=45461, storageInfo=lv=-57;cid=testClusterID;nsid=313737442;c=1689318399097), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 07:06:41,233 DEBUG [Listener at localhost/46223] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b 2023-07-14 07:06:41,295 INFO [Listener at localhost/46223] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278/zookeeper_0, clientPort=57240, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-14 07:06:41,309 INFO [Listener at localhost/46223] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57240 2023-07-14 07:06:41,321 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:41,324 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:41,991 INFO [Listener at localhost/46223] util.FSUtils(471): Created version file at hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d with version=8 2023-07-14 07:06:41,991 INFO [Listener at localhost/46223] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/hbase-staging 2023-07-14 07:06:42,316 INFO [Listener at localhost/46223] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-14 07:06:42,787 INFO [Listener at localhost/46223] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-14 07:06:42,819 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:42,819 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:42,820 INFO [Listener at localhost/46223] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 07:06:42,820 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:42,820 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 07:06:42,960 INFO [Listener at localhost/46223] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 07:06:43,033 DEBUG [Listener at localhost/46223] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-14 07:06:43,127 INFO [Listener at localhost/46223] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41343 2023-07-14 07:06:43,137 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:43,139 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:43,162 INFO [Listener at localhost/46223] zookeeper.RecoverableZooKeeper(93): Process identifier=master:41343 connecting to ZooKeeper ensemble=127.0.0.1:57240 2023-07-14 07:06:43,203 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:413430x0, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 07:06:43,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:41343-0x10162a7a8040000 connected 2023-07-14 07:06:43,231 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 07:06:43,231 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 07:06:43,235 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 07:06:43,242 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41343 2023-07-14 07:06:43,242 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41343 2023-07-14 07:06:43,243 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41343 2023-07-14 07:06:43,243 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41343 2023-07-14 07:06:43,244 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41343 2023-07-14 07:06:43,249 INFO [Listener at localhost/46223] master.HMaster(444): hbase.rootdir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d, hbase.cluster.distributed=false 2023-07-14 07:06:43,317 INFO [Listener at localhost/46223] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-14 07:06:43,317 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:43,317 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:43,318 INFO [Listener at localhost/46223] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 07:06:43,318 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:43,318 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 07:06:43,323 INFO [Listener at localhost/46223] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 07:06:43,326 INFO [Listener at localhost/46223] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33537 2023-07-14 07:06:43,328 INFO [Listener at localhost/46223] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 07:06:43,333 DEBUG [Listener at localhost/46223] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 07:06:43,335 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:43,336 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:43,338 INFO [Listener at localhost/46223] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33537 connecting to ZooKeeper ensemble=127.0.0.1:57240 2023-07-14 07:06:43,341 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:335370x0, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 07:06:43,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33537-0x10162a7a8040001 connected 2023-07-14 07:06:43,342 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 07:06:43,343 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 07:06:43,344 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 07:06:43,345 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33537 2023-07-14 07:06:43,345 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33537 2023-07-14 07:06:43,345 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33537 2023-07-14 07:06:43,346 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33537 2023-07-14 07:06:43,347 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33537 2023-07-14 07:06:43,359 INFO [Listener at localhost/46223] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-14 07:06:43,359 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:43,360 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:43,360 INFO [Listener at localhost/46223] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 07:06:43,360 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:43,360 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 07:06:43,360 INFO [Listener at localhost/46223] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 07:06:43,362 INFO [Listener at localhost/46223] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45707 2023-07-14 07:06:43,362 INFO [Listener at localhost/46223] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 07:06:43,364 DEBUG [Listener at localhost/46223] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 07:06:43,364 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:43,366 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:43,368 INFO [Listener at localhost/46223] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45707 connecting to ZooKeeper ensemble=127.0.0.1:57240 2023-07-14 07:06:43,372 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:457070x0, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 07:06:43,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:45707-0x10162a7a8040002 connected 2023-07-14 07:06:43,373 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 07:06:43,373 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 07:06:43,374 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 07:06:43,375 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45707 2023-07-14 07:06:43,375 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45707 2023-07-14 07:06:43,375 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45707 2023-07-14 07:06:43,376 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45707 2023-07-14 07:06:43,376 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45707 2023-07-14 07:06:43,377 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:06:43,386 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-14 07:06:43,387 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:06:43,405 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 07:06:43,405 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 07:06:43,405 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 07:06:43,405 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:06:43,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 07:06:43,408 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,41343,1689318402146 from backup master directory 2023-07-14 07:06:43,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 07:06:43,411 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:06:43,411 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-14 07:06:43,412 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 07:06:43,412 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:06:43,414 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-14 07:06:43,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-14 07:06:43,504 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/hbase.id with ID: 2fc4a3dd-1b64-4a36-a4a9-8269119c0623 2023-07-14 07:06:43,546 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:43,561 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:06:43,602 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x3d673313 to 127.0.0.1:57240 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 07:06:43,634 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56bd0c79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 07:06:43,657 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-14 07:06:43,659 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-14 07:06:43,676 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-14 07:06:43,676 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-14 07:06:43,678 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-14 07:06:43,682 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-14 07:06:43,683 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 07:06:43,717 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/data/master/store-tmp 2023-07-14 07:06:43,748 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:06:43,749 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-14 07:06:43,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 07:06:43,749 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 07:06:43,749 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-14 07:06:43,749 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 07:06:43,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 07:06:43,749 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 07:06:43,750 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/WALs/jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:06:43,771 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41343%2C1689318402146, suffix=, logDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/WALs/jenkins-hbase4.apache.org,41343,1689318402146, archiveDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/oldWALs, maxLogs=10 2023-07-14 07:06:43,826 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK] 2023-07-14 07:06:43,826 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK] 2023-07-14 07:06:43,834 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 07:06:43,899 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/WALs/jenkins-hbase4.apache.org,41343,1689318402146/jenkins-hbase4.apache.org%2C41343%2C1689318402146.1689318403781 2023-07-14 07:06:43,900 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK], DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK]] 2023-07-14 07:06:43,900 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:06:43,901 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:06:43,904 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 07:06:43,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 07:06:43,960 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-14 07:06:43,967 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-14 07:06:43,990 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-14 07:06:44,001 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:44,009 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-14 07:06:44,010 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-14 07:06:44,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 07:06:44,033 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 07:06:44,035 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74858973, jitterRate=0.11548562347888947}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 07:06:44,035 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 07:06:44,036 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-14 07:06:44,055 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-14 07:06:44,056 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-14 07:06:44,058 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-14 07:06:44,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-14 07:06:44,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-14 07:06:44,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-14 07:06:44,116 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-14 07:06:44,122 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-14 07:06:44,148 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-14 07:06:44,152 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-14 07:06:44,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-14 07:06:44,158 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-14 07:06:44,162 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-14 07:06:44,165 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:06:44,166 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-14 07:06:44,167 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-14 07:06:44,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-14 07:06:44,183 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 07:06:44,183 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 07:06:44,183 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 07:06:44,183 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:06:44,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,41343,1689318402146, sessionid=0x10162a7a8040000, setting cluster-up flag (Was=false) 2023-07-14 07:06:44,199 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:06:44,206 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-14 07:06:44,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:06:44,212 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:06:44,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-14 07:06:44,219 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:06:44,222 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.hbase-snapshot/.tmp 2023-07-14 07:06:44,279 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(951): ClusterId : 2fc4a3dd-1b64-4a36-a4a9-8269119c0623 2023-07-14 07:06:44,279 INFO [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(951): ClusterId : 2fc4a3dd-1b64-4a36-a4a9-8269119c0623 2023-07-14 07:06:44,286 DEBUG [RS:0;jenkins-hbase4:33537] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 07:06:44,286 DEBUG [RS:1;jenkins-hbase4:45707] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 07:06:44,293 DEBUG [RS:1;jenkins-hbase4:45707] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 07:06:44,293 DEBUG [RS:1;jenkins-hbase4:45707] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 07:06:44,295 DEBUG [RS:0;jenkins-hbase4:33537] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 07:06:44,295 DEBUG [RS:0;jenkins-hbase4:33537] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 07:06:44,298 DEBUG [RS:0;jenkins-hbase4:33537] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 07:06:44,298 DEBUG [RS:1;jenkins-hbase4:45707] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 07:06:44,299 DEBUG [RS:0;jenkins-hbase4:33537] zookeeper.ReadOnlyZKClient(139): Connect 0x23c63626 to 127.0.0.1:57240 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 07:06:44,300 DEBUG [RS:1;jenkins-hbase4:45707] zookeeper.ReadOnlyZKClient(139): Connect 0x2d1dccec to 127.0.0.1:57240 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 07:06:44,306 DEBUG [RS:0;jenkins-hbase4:33537] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d01cc2e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 07:06:44,307 DEBUG [RS:1;jenkins-hbase4:45707] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23da373f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 07:06:44,307 DEBUG [RS:0;jenkins-hbase4:33537] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@293e5d56, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-14 07:06:44,307 DEBUG [RS:1;jenkins-hbase4:45707] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27e5a79c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-14 07:06:44,331 DEBUG [RS:0;jenkins-hbase4:33537] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:33537 2023-07-14 07:06:44,331 DEBUG [RS:1;jenkins-hbase4:45707] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:45707 2023-07-14 07:06:44,336 INFO [RS:0;jenkins-hbase4:33537] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 07:06:44,336 INFO [RS:1;jenkins-hbase4:45707] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 07:06:44,336 INFO [RS:1;jenkins-hbase4:45707] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 07:06:44,336 INFO [RS:0;jenkins-hbase4:33537] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 07:06:44,336 DEBUG [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 07:06:44,337 DEBUG [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 07:06:44,339 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,41343,1689318402146 with isa=jenkins-hbase4.apache.org/172.31.14.131:33537, startcode=1689318403316 2023-07-14 07:06:44,339 INFO [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,41343,1689318402146 with isa=jenkins-hbase4.apache.org/172.31.14.131:45707, startcode=1689318403359 2023-07-14 07:06:44,344 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-14 07:06:44,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-14 07:06:44,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-14 07:06:44,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-14 07:06:44,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-14 07:06:44,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-14 07:06:44,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-14 07:06:44,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689318434357 2023-07-14 07:06:44,357 DEBUG [RS:1;jenkins-hbase4:45707] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 07:06:44,357 DEBUG [RS:0;jenkins-hbase4:33537] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 07:06:44,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-14 07:06:44,363 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-14 07:06:44,363 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-14 07:06:44,368 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-14 07:06:44,369 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-14 07:06:44,377 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-14 07:06:44,378 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-14 07:06:44,378 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-14 07:06:44,378 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-14 07:06:44,379 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,381 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-14 07:06:44,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-14 07:06:44,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-14 07:06:44,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-14 07:06:44,388 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-14 07:06:44,389 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689318404389,5,FailOnTimeoutGroup] 2023-07-14 07:06:44,389 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689318404389,5,FailOnTimeoutGroup] 2023-07-14 07:06:44,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-14 07:06:44,391 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,392 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,415 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-14 07:06:44,416 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-14 07:06:44,417 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d 2023-07-14 07:06:44,423 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39627, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 07:06:44,423 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60021, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 07:06:44,438 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:06:44,440 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41343] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:44,441 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 07:06:44,443 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/info 2023-07-14 07:06:44,444 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 07:06:44,445 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:44,446 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 07:06:44,448 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/rep_barrier 2023-07-14 07:06:44,449 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 07:06:44,450 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:44,450 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 07:06:44,451 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41343] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:44,454 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/table 2023-07-14 07:06:44,455 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 07:06:44,456 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:44,459 DEBUG [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d 2023-07-14 07:06:44,459 DEBUG [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d 2023-07-14 07:06:44,459 DEBUG [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:35789 2023-07-14 07:06:44,459 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740 2023-07-14 07:06:44,459 DEBUG [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 07:06:44,459 DEBUG [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:35789 2023-07-14 07:06:44,459 DEBUG [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 07:06:44,460 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740 2023-07-14 07:06:44,464 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 07:06:44,465 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:06:44,467 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 07:06:44,468 DEBUG [RS:1;jenkins-hbase4:45707] zookeeper.ZKUtil(162): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:44,469 WARN [RS:1;jenkins-hbase4:45707] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 07:06:44,469 INFO [RS:1;jenkins-hbase4:45707] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 07:06:44,470 DEBUG [RS:0;jenkins-hbase4:33537] zookeeper.ZKUtil(162): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:44,470 DEBUG [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1948): logDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:44,471 WARN [RS:0;jenkins-hbase4:33537] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 07:06:44,471 INFO [RS:0;jenkins-hbase4:33537] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 07:06:44,472 DEBUG [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1948): logDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:44,472 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,45707,1689318403359] 2023-07-14 07:06:44,472 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33537,1689318403316] 2023-07-14 07:06:44,474 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 07:06:44,476 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70268070, jitterRate=0.04707583785057068}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 07:06:44,477 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 07:06:44,477 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 07:06:44,477 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 07:06:44,477 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 07:06:44,477 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 07:06:44,478 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 07:06:44,478 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 07:06:44,479 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 07:06:44,482 DEBUG [RS:1;jenkins-hbase4:45707] zookeeper.ZKUtil(162): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:44,482 DEBUG [RS:0;jenkins-hbase4:33537] zookeeper.ZKUtil(162): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:44,483 DEBUG [RS:0;jenkins-hbase4:33537] zookeeper.ZKUtil(162): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:44,483 DEBUG [RS:1;jenkins-hbase4:45707] zookeeper.ZKUtil(162): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:44,484 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-14 07:06:44,484 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-14 07:06:44,493 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-14 07:06:44,494 DEBUG [RS:1;jenkins-hbase4:45707] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 07:06:44,494 DEBUG [RS:0;jenkins-hbase4:33537] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 07:06:44,504 INFO [RS:0;jenkins-hbase4:33537] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 07:06:44,504 INFO [RS:1;jenkins-hbase4:45707] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 07:06:44,506 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-14 07:06:44,509 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-14 07:06:44,526 INFO [RS:1;jenkins-hbase4:45707] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 07:06:44,526 INFO [RS:0;jenkins-hbase4:33537] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 07:06:44,530 INFO [RS:1;jenkins-hbase4:45707] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 07:06:44,530 INFO [RS:0;jenkins-hbase4:33537] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 07:06:44,530 INFO [RS:1;jenkins-hbase4:45707] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,530 INFO [RS:0;jenkins-hbase4:33537] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,531 INFO [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 07:06:44,531 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 07:06:44,539 INFO [RS:1;jenkins-hbase4:45707] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,539 INFO [RS:0;jenkins-hbase4:33537] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,539 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,539 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,539 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-14 07:06:44,540 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-14 07:06:44,540 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,540 DEBUG [RS:1;jenkins-hbase4:45707] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,541 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,541 DEBUG [RS:0;jenkins-hbase4:33537] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:44,542 INFO [RS:1;jenkins-hbase4:45707] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,542 INFO [RS:1;jenkins-hbase4:45707] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,542 INFO [RS:1;jenkins-hbase4:45707] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,542 INFO [RS:0;jenkins-hbase4:33537] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,542 INFO [RS:0;jenkins-hbase4:33537] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,542 INFO [RS:0;jenkins-hbase4:33537] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,566 INFO [RS:1;jenkins-hbase4:45707] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 07:06:44,566 INFO [RS:0;jenkins-hbase4:33537] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 07:06:44,569 INFO [RS:1;jenkins-hbase4:45707] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45707,1689318403359-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,569 INFO [RS:0;jenkins-hbase4:33537] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33537,1689318403316-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:44,587 INFO [RS:0;jenkins-hbase4:33537] regionserver.Replication(203): jenkins-hbase4.apache.org,33537,1689318403316 started 2023-07-14 07:06:44,587 INFO [RS:1;jenkins-hbase4:45707] regionserver.Replication(203): jenkins-hbase4.apache.org,45707,1689318403359 started 2023-07-14 07:06:44,587 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,33537,1689318403316, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33537, sessionid=0x10162a7a8040001 2023-07-14 07:06:44,587 INFO [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,45707,1689318403359, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:45707, sessionid=0x10162a7a8040002 2023-07-14 07:06:44,587 DEBUG [RS:1;jenkins-hbase4:45707] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 07:06:44,587 DEBUG [RS:0;jenkins-hbase4:33537] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 07:06:44,587 DEBUG [RS:1;jenkins-hbase4:45707] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:44,587 DEBUG [RS:0;jenkins-hbase4:33537] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:44,588 DEBUG [RS:1;jenkins-hbase4:45707] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45707,1689318403359' 2023-07-14 07:06:44,588 DEBUG [RS:0;jenkins-hbase4:33537] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33537,1689318403316' 2023-07-14 07:06:44,589 DEBUG [RS:0;jenkins-hbase4:33537] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 07:06:44,589 DEBUG [RS:1;jenkins-hbase4:45707] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 07:06:44,589 DEBUG [RS:1;jenkins-hbase4:45707] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 07:06:44,589 DEBUG [RS:0;jenkins-hbase4:33537] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 07:06:44,590 DEBUG [RS:1;jenkins-hbase4:45707] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 07:06:44,590 DEBUG [RS:0;jenkins-hbase4:33537] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 07:06:44,590 DEBUG [RS:1;jenkins-hbase4:45707] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 07:06:44,590 DEBUG [RS:0;jenkins-hbase4:33537] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 07:06:44,590 DEBUG [RS:1;jenkins-hbase4:45707] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:44,590 DEBUG [RS:0;jenkins-hbase4:33537] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:44,591 DEBUG [RS:0;jenkins-hbase4:33537] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33537,1689318403316' 2023-07-14 07:06:44,591 DEBUG [RS:0;jenkins-hbase4:33537] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 07:06:44,591 DEBUG [RS:1;jenkins-hbase4:45707] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45707,1689318403359' 2023-07-14 07:06:44,591 DEBUG [RS:1;jenkins-hbase4:45707] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 07:06:44,591 DEBUG [RS:1;jenkins-hbase4:45707] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 07:06:44,591 DEBUG [RS:0;jenkins-hbase4:33537] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 07:06:44,592 DEBUG [RS:1;jenkins-hbase4:45707] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 07:06:44,592 INFO [RS:1;jenkins-hbase4:45707] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 07:06:44,592 INFO [RS:1;jenkins-hbase4:45707] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 07:06:44,592 DEBUG [RS:0;jenkins-hbase4:33537] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 07:06:44,593 INFO [RS:0;jenkins-hbase4:33537] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 07:06:44,593 INFO [RS:0;jenkins-hbase4:33537] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 07:06:44,661 DEBUG [jenkins-hbase4:41343] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-14 07:06:44,665 DEBUG [jenkins-hbase4:41343] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-14 07:06:44,671 DEBUG [jenkins-hbase4:41343] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 07:06:44,671 DEBUG [jenkins-hbase4:41343] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 07:06:44,671 DEBUG [jenkins-hbase4:41343] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 07:06:44,674 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33537,1689318403316, state=OPENING 2023-07-14 07:06:44,681 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-14 07:06:44,685 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:06:44,686 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 07:06:44,689 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33537,1689318403316}] 2023-07-14 07:06:44,704 INFO [RS:0;jenkins-hbase4:33537] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33537%2C1689318403316, suffix=, logDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316, archiveDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs, maxLogs=32 2023-07-14 07:06:44,704 INFO [RS:1;jenkins-hbase4:45707] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45707%2C1689318403359, suffix=, logDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359, archiveDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs, maxLogs=32 2023-07-14 07:06:44,728 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK] 2023-07-14 07:06:44,728 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK] 2023-07-14 07:06:44,728 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK] 2023-07-14 07:06:44,729 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK] 2023-07-14 07:06:44,738 INFO [RS:1;jenkins-hbase4:45707] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709 2023-07-14 07:06:44,738 INFO [RS:0;jenkins-hbase4:33537] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709 2023-07-14 07:06:44,738 DEBUG [RS:1;jenkins-hbase4:45707] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK], DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK]] 2023-07-14 07:06:44,738 DEBUG [RS:0;jenkins-hbase4:33537] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK], DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK]] 2023-07-14 07:06:44,876 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:44,878 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 07:06:44,881 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42218, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 07:06:44,893 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-14 07:06:44,893 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 07:06:44,897 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta, suffix=.meta, logDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316, archiveDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs, maxLogs=32 2023-07-14 07:06:44,917 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK] 2023-07-14 07:06:44,917 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK] 2023-07-14 07:06:44,924 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta 2023-07-14 07:06:44,925 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK], DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK]] 2023-07-14 07:06:44,926 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:06:44,927 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-14 07:06:44,943 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-14 07:06:44,974 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-14 07:06:44,979 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-14 07:06:44,980 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:06:44,980 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-14 07:06:44,980 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-14 07:06:44,983 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 07:06:44,985 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/info 2023-07-14 07:06:44,985 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/info 2023-07-14 07:06:44,985 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 07:06:44,986 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:44,986 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 07:06:44,988 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/rep_barrier 2023-07-14 07:06:44,988 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/rep_barrier 2023-07-14 07:06:44,988 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 07:06:44,989 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:44,989 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 07:06:44,991 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/table 2023-07-14 07:06:44,991 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/table 2023-07-14 07:06:44,991 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 07:06:44,992 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:44,994 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740 2023-07-14 07:06:44,996 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740 2023-07-14 07:06:44,999 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 07:06:45,002 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 07:06:45,003 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70466184, jitterRate=0.05002796649932861}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 07:06:45,003 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 07:06:45,013 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689318404869 2023-07-14 07:06:45,030 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-14 07:06:45,031 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-14 07:06:45,031 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33537,1689318403316, state=OPEN 2023-07-14 07:06:45,034 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 07:06:45,034 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 07:06:45,038 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-14 07:06:45,038 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33537,1689318403316 in 345 msec 2023-07-14 07:06:45,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-14 07:06:45,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 547 msec 2023-07-14 07:06:45,048 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 780 msec 2023-07-14 07:06:45,048 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689318405048, completionTime=-1 2023-07-14 07:06:45,049 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-14 07:06:45,049 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-14 07:06:45,111 DEBUG [hconnection-0x42741d3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 07:06:45,114 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42222, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 07:06:45,129 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-14 07:06:45,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689318465130 2023-07-14 07:06:45,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689318525130 2023-07-14 07:06:45,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 80 msec 2023-07-14 07:06:45,152 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41343,1689318402146-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:45,152 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41343,1689318402146-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:45,152 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41343,1689318402146-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:45,154 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:41343, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:45,154 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:45,160 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-14 07:06:45,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-14 07:06:45,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-14 07:06:45,179 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-14 07:06:45,181 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-14 07:06:45,184 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-14 07:06:45,205 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:06:45,207 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f empty. 2023-07-14 07:06:45,208 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:06:45,208 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-14 07:06:45,236 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-14 07:06:45,238 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7e77fde02849b028ee4ee8683f32fb6f, NAME => 'hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp 2023-07-14 07:06:45,255 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:06:45,255 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 7e77fde02849b028ee4ee8683f32fb6f, disabling compactions & flushes 2023-07-14 07:06:45,255 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:06:45,255 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:06:45,255 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. after waiting 0 ms 2023-07-14 07:06:45,255 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:06:45,255 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:06:45,255 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 7e77fde02849b028ee4ee8683f32fb6f: 2023-07-14 07:06:45,259 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-14 07:06:45,277 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689318405262"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689318405262"}]},"ts":"1689318405262"} 2023-07-14 07:06:45,308 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-14 07:06:45,310 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-14 07:06:45,315 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689318405311"}]},"ts":"1689318405311"} 2023-07-14 07:06:45,319 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-14 07:06:45,325 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-14 07:06:45,326 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 07:06:45,326 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 07:06:45,326 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 07:06:45,329 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7e77fde02849b028ee4ee8683f32fb6f, ASSIGN}] 2023-07-14 07:06:45,332 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7e77fde02849b028ee4ee8683f32fb6f, ASSIGN 2023-07-14 07:06:45,335 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7e77fde02849b028ee4ee8683f32fb6f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33537,1689318403316; forceNewPlan=false, retain=false 2023-07-14 07:06:45,487 INFO [jenkins-hbase4:41343] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-14 07:06:45,488 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7e77fde02849b028ee4ee8683f32fb6f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:45,488 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689318405488"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689318405488"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689318405488"}]},"ts":"1689318405488"} 2023-07-14 07:06:45,494 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 7e77fde02849b028ee4ee8683f32fb6f, server=jenkins-hbase4.apache.org,33537,1689318403316}] 2023-07-14 07:06:45,654 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:06:45,655 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7e77fde02849b028ee4ee8683f32fb6f, NAME => 'hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:06:45,656 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:06:45,656 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:06:45,656 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:06:45,656 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:06:45,658 INFO [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:06:45,660 DEBUG [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/info 2023-07-14 07:06:45,660 DEBUG [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/info 2023-07-14 07:06:45,661 INFO [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7e77fde02849b028ee4ee8683f32fb6f columnFamilyName info 2023-07-14 07:06:45,662 INFO [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] regionserver.HStore(310): Store=7e77fde02849b028ee4ee8683f32fb6f/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:45,663 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:06:45,664 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:06:45,669 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:06:45,671 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 07:06:45,672 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7e77fde02849b028ee4ee8683f32fb6f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73096615, jitterRate=0.0892244428396225}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 07:06:45,672 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7e77fde02849b028ee4ee8683f32fb6f: 2023-07-14 07:06:45,674 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f., pid=6, masterSystemTime=1689318405648 2023-07-14 07:06:45,678 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:06:45,678 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:06:45,679 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7e77fde02849b028ee4ee8683f32fb6f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:45,680 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689318405679"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689318405679"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689318405679"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689318405679"}]},"ts":"1689318405679"} 2023-07-14 07:06:45,687 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-14 07:06:45,687 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 7e77fde02849b028ee4ee8683f32fb6f, server=jenkins-hbase4.apache.org,33537,1689318403316 in 190 msec 2023-07-14 07:06:45,691 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-14 07:06:45,691 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7e77fde02849b028ee4ee8683f32fb6f, ASSIGN in 358 msec 2023-07-14 07:06:45,692 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-14 07:06:45,693 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689318405692"}]},"ts":"1689318405692"} 2023-07-14 07:06:45,695 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-14 07:06:45,699 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-14 07:06:45,702 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 528 msec 2023-07-14 07:06:45,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-14 07:06:45,784 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-14 07:06:45,785 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:06:45,818 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-14 07:06:45,835 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-14 07:06:45,842 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-07-14 07:06:45,851 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-14 07:06:45,862 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-14 07:06:45,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-07-14 07:06:45,877 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-14 07:06:45,880 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-14 07:06:45,880 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.468sec 2023-07-14 07:06:45,882 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-14 07:06:45,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-14 07:06:45,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-14 07:06:45,885 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41343,1689318402146-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-14 07:06:45,886 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41343,1689318402146-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-14 07:06:45,894 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-14 07:06:45,985 DEBUG [Listener at localhost/46223] zookeeper.ReadOnlyZKClient(139): Connect 0x65b7854e to 127.0.0.1:57240 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 07:06:45,991 DEBUG [Listener at localhost/46223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@707a7b2d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 07:06:46,005 DEBUG [hconnection-0x4e06eedf-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 07:06:46,017 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 07:06:46,027 INFO [Listener at localhost/46223] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:06:46,038 DEBUG [Listener at localhost/46223] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-14 07:06:46,042 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54546, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-14 07:06:46,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-14 07:06:46,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-14 07:06:46,059 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-14 07:06:46,061 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-14 07:06:46,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-14 07:06:46,064 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp/data/default/t/10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,065 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp/data/default/t/10a151fabcc53ca1e27f811541e9729d empty. 2023-07-14 07:06:46,067 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp/data/default/t/10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,067 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-14 07:06:46,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 07:06:46,086 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-14 07:06:46,087 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 10a151fabcc53ca1e27f811541e9729d, NAME => 't,,1689318406051.10a151fabcc53ca1e27f811541e9729d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/.tmp 2023-07-14 07:06:46,102 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:06:46,102 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 10a151fabcc53ca1e27f811541e9729d, disabling compactions & flushes 2023-07-14 07:06:46,102 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,102 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,102 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. after waiting 0 ms 2023-07-14 07:06:46,102 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,102 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,102 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 10a151fabcc53ca1e27f811541e9729d: 2023-07-14 07:06:46,106 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-14 07:06:46,108 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689318406107"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689318406107"}]},"ts":"1689318406107"} 2023-07-14 07:06:46,110 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-14 07:06:46,111 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-14 07:06:46,112 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689318406111"}]},"ts":"1689318406111"} 2023-07-14 07:06:46,114 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-14 07:06:46,118 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-14 07:06:46,118 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 07:06:46,119 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 07:06:46,119 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 07:06:46,119 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN}] 2023-07-14 07:06:46,121 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN 2023-07-14 07:06:46,122 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45707,1689318403359; forceNewPlan=false, retain=false 2023-07-14 07:06:46,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 07:06:46,272 INFO [jenkins-hbase4:41343] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-14 07:06:46,273 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=10a151fabcc53ca1e27f811541e9729d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:46,273 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689318406273"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689318406273"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689318406273"}]},"ts":"1689318406273"} 2023-07-14 07:06:46,277 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 10a151fabcc53ca1e27f811541e9729d, server=jenkins-hbase4.apache.org,45707,1689318403359}] 2023-07-14 07:06:46,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 07:06:46,430 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:46,431 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 07:06:46,433 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53518, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 07:06:46,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 10a151fabcc53ca1e27f811541e9729d, NAME => 't,,1689318406051.10a151fabcc53ca1e27f811541e9729d.', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:06:46,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:06:46,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,442 INFO [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,444 DEBUG [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/f 2023-07-14 07:06:46,444 DEBUG [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/f 2023-07-14 07:06:46,445 INFO [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 10a151fabcc53ca1e27f811541e9729d columnFamilyName f 2023-07-14 07:06:46,446 INFO [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] regionserver.HStore(310): Store=10a151fabcc53ca1e27f811541e9729d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:46,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 07:06:46,457 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 10a151fabcc53ca1e27f811541e9729d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66911957, jitterRate=-0.0029341429471969604}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 07:06:46,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 10a151fabcc53ca1e27f811541e9729d: 2023-07-14 07:06:46,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689318406051.10a151fabcc53ca1e27f811541e9729d., pid=11, masterSystemTime=1689318406430 2023-07-14 07:06:46,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,464 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=10a151fabcc53ca1e27f811541e9729d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:46,465 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689318406464"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689318406464"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689318406464"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689318406464"}]},"ts":"1689318406464"} 2023-07-14 07:06:46,471 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-14 07:06:46,471 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 10a151fabcc53ca1e27f811541e9729d, server=jenkins-hbase4.apache.org,45707,1689318403359 in 191 msec 2023-07-14 07:06:46,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-14 07:06:46,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN in 352 msec 2023-07-14 07:06:46,475 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-14 07:06:46,475 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689318406475"}]},"ts":"1689318406475"} 2023-07-14 07:06:46,477 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-14 07:06:46,480 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-14 07:06:46,483 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 425 msec 2023-07-14 07:06:46,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 07:06:46,683 INFO [Listener at localhost/46223] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-14 07:06:46,683 DEBUG [Listener at localhost/46223] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-14 07:06:46,685 INFO [Listener at localhost/46223] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-14 07:06:46,689 INFO [Listener at localhost/46223] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-14 07:06:46,689 INFO [Listener at localhost/46223] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-14 07:06:46,689 INFO [Listener at localhost/46223] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-14 07:06:46,692 DEBUG [Listener at localhost/46223] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 07:06:46,695 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53530, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 07:06:46,738 INFO [Listener at localhost/46223] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=295, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=102, ProcessCount=171, AvailableMemoryMB=7036 2023-07-14 07:06:46,740 INFO [Listener at localhost/46223] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-14 07:06:46,743 DEBUG [Listener at localhost/46223] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 07:06:46,744 DEBUG [Listener at localhost/46223] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,744 INFO [Listener at localhost/46223] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-14 07:06:46,744 INFO [Listener at localhost/46223] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-14 07:06:46,747 INFO [Listener at localhost/46223] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689318405692/Put/vlen=2/seqid=0} 2023-07-14 07:06:46,748 INFO [Listener at localhost/46223] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-14 07:06:46,748 INFO [Listener at localhost/46223] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f./info:regioninfo/1689318405679/Put/vlen=41/seqid=0, hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f./info:seqnumDuringOpen/1689318405679/Put/vlen=8/seqid=0, hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f./info:server/1689318405679/Put/vlen=31/seqid=0, hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f./info:serverstartcode/1689318405679/Put/vlen=8/seqid=0, hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f./info:sn/1689318405488/Put/vlen=45/seqid=0, hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f./info:state/1689318405679/Put/vlen=4/seqid=0} 2023-07-14 07:06:46,748 INFO [Listener at localhost/46223] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 7e77fde02849b028ee4ee8683f32fb6f, NAME => 'hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:06:46,748 INFO [Listener at localhost/46223] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689318406475/Put/vlen=2/seqid=0} 2023-07-14 07:06:46,748 INFO [Listener at localhost/46223] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-14 07:06:46,748 INFO [Listener at localhost/46223] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689318406051.10a151fabcc53ca1e27f811541e9729d./info:regioninfo/1689318406464/Put/vlen=35/seqid=0, t,,1689318406051.10a151fabcc53ca1e27f811541e9729d./info:seqnumDuringOpen/1689318406464/Put/vlen=8/seqid=0, t,,1689318406051.10a151fabcc53ca1e27f811541e9729d./info:server/1689318406464/Put/vlen=31/seqid=0, t,,1689318406051.10a151fabcc53ca1e27f811541e9729d./info:serverstartcode/1689318406464/Put/vlen=8/seqid=0, t,,1689318406051.10a151fabcc53ca1e27f811541e9729d./info:sn/1689318406273/Put/vlen=45/seqid=0, t,,1689318406051.10a151fabcc53ca1e27f811541e9729d./info:state/1689318406464/Put/vlen=4/seqid=0} 2023-07-14 07:06:46,749 INFO [Listener at localhost/46223] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 10a151fabcc53ca1e27f811541e9729d, NAME => 't,,1689318406051.10a151fabcc53ca1e27f811541e9729d.', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:06:46,749 DEBUG [Listener at localhost/46223] zookeeper.ReadOnlyZKClient(139): Connect 0x0700f054 to 127.0.0.1:57240 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 07:06:46,755 DEBUG [Listener at localhost/46223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4147d733, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 07:06:46,756 INFO [Listener at localhost/46223] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-14 07:06:46,756 INFO [Listener at localhost/46223] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-14 07:06:46,758 DEBUG [hconnection-0x1ef800db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 07:06:46,761 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42240, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 07:06:46,764 DEBUG [hconnection-0x1ef800db-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 07:06:46,766 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 07:06:46,768 INFO [Listener at localhost/46223] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-14 07:06:46,771 ERROR [Listener at localhost/46223] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,45707,1689318403359: die! ***** 2023-07-14 07:06:46,772 ERROR [Listener at localhost/46223] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-14 07:06:46,775 DEBUG [Listener at localhost/46223] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-14 07:06:46,777 DEBUG [Listener at localhost/46223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-14 07:06:46,778 DEBUG [Listener at localhost/46223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-14 07:06:46,778 DEBUG [Listener at localhost/46223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-14 07:06:46,779 INFO [Listener at localhost/46223] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1135607808, "init": 513802240, "max": 2051014656, "used": 459860440 }, "NonHeapMemoryUsage": { "committed": 112746496, "init": 2555904, "max": -1, "used": 110670144 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-14 07:06:46,784 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41343] master.MasterRpcServices(612): jenkins-hbase4.apache.org,45707,1689318403359 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,45707,1689318403359: die! ***** 2023-07-14 07:06:46,786 INFO [Listener at localhost/46223] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,45707,1689318403359' ***** 2023-07-14 07:06:46,787 INFO [Listener at localhost/46223] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-14 07:06:46,787 INFO [RS:1;jenkins-hbase4:45707] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 07:06:46,787 INFO [RS:1;jenkins-hbase4:45707] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-14 07:06:46,787 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 07:06:46,788 INFO [RS:1;jenkins-hbase4:45707] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-14 07:06:46,788 INFO [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(3305): Received CLOSE for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:46,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45707] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -6131779393791459327 2023-07-14 07:06:46,788 INFO [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:46,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45707] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45707,1689318403359 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 07:06:46,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 10a151fabcc53ca1e27f811541e9729d, disabling compactions & flushes 2023-07-14 07:06:46,789 DEBUG [RS:1;jenkins-hbase4:45707] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d1dccec to 127.0.0.1:57240 2023-07-14 07:06:46,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45707] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:53540 deadline: 1689318466788, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45707,1689318403359 aborting 2023-07-14 07:06:46,789 DEBUG [RS:1;jenkins-hbase4:45707] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:06:46,789 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,790 INFO [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-14 07:06:46,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. after waiting 0 ms 2023-07-14 07:06:46,790 DEBUG [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1478): Online Regions={10a151fabcc53ca1e27f811541e9729d=t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.} 2023-07-14 07:06:46,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,791 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,791 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 10a151fabcc53ca1e27f811541e9729d: 2023-07-14 07:06:46,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:46,792 DEBUG [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1504): Waiting on 2023-07-14 07:06:46,806 DEBUG [Listener at localhost/46223] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45707,1689318403359 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45707,1689318403359 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-14 07:06:46,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45707] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45707,1689318403359 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 07:06:46,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45707] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:53540 deadline: 1689318466808, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45707,1689318403359 aborting 2023-07-14 07:06:46,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45707] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45707,1689318403359 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 07:06:46,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45707] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:53540 deadline: 1689318466918, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45707,1689318403359 aborting 2023-07-14 07:06:46,992 INFO [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,45707,1689318403359; all regions closed. 2023-07-14 07:06:47,002 DEBUG [RS:1;jenkins-hbase4:45707] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:06:47,002 INFO [RS:1;jenkins-hbase4:45707] regionserver.LeaseManager(133): Closed leases 2023-07-14 07:06:47,003 INFO [RS:1;jenkins-hbase4:45707] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-14 07:06:47,003 INFO [RS:1;jenkins-hbase4:45707] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 07:06:47,003 INFO [RS:1;jenkins-hbase4:45707] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 07:06:47,003 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 07:06:47,003 INFO [RS:1;jenkins-hbase4:45707] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 07:06:47,004 INFO [RS:1;jenkins-hbase4:45707] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45707 2023-07-14 07:06:47,013 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:47,013 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:06:47,013 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:47,013 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:06:47,013 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:06:47,015 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,45707,1689318403359] 2023-07-14 07:06:47,015 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,45707,1689318403359; numProcessing=1 2023-07-14 07:06:47,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:47,016 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,45707,1689318403359 znode expired, triggering replicatorRemoved event 2023-07-14 07:06:47,017 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,45707,1689318403359 already deleted, retry=false 2023-07-14 07:06:47,017 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,45707,1689318403359 on jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:06:47,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:47,025 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,45707,1689318403359, splitWal=true, meta=false 2023-07-14 07:06:47,025 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,45707,1689318403359 (carryingMeta=false) jenkins-hbase4.apache.org,45707,1689318403359/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@7e618cfd[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-14 07:06:47,028 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45707,1689318403359, splitWal=true, meta=false 2023-07-14 07:06:47,029 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,45707,1689318403359 had 1 regions 2023-07-14 07:06:47,031 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45707,1689318403359, splitWal=true, meta=false, isMeta: false 2023-07-14 07:06:47,033 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting 2023-07-14 07:06:47,034 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,45707,1689318403359 WAL count=1, meta=false 2023-07-14 07:06:47,037 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709}] 2023-07-14 07:06:47,045 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:47,046 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709, worker=jenkins-hbase4.apache.org,33537,1689318403316}] 2023-07-14 07:06:47,115 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 07:06:47,116 INFO [RS:1;jenkins-hbase4:45707] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,45707,1689318403359; zookeeper connection closed. 2023-07-14 07:06:47,116 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:45707-0x10162a7a8040002, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 07:06:47,116 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@278f001c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@278f001c 2023-07-14 07:06:47,125 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:45707 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45707 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 07:06:47,126 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:45707 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45707 2023-07-14 07:06:47,210 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33537] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-14 07:06:47,233 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709, size=1.1 K (1148bytes) 2023-07-14 07:06:47,234 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709 2023-07-14 07:06:47,238 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709 after 4ms 2023-07-14 07:06:47,244 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709 took 13ms 2023-07-14 07:06:47,259 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 10a151fabcc53ca1e27f811541e9729d: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-14 07:06:47,261 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709 so closing down 2023-07-14 07:06:47,261 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 07:06:47,267 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-14 07:06:47,269 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709.temp 2023-07-14 07:06:47,270 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 07:06:47,281 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-14 07:06:47,283 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709.temp to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000013 2023-07-14 07:06:47,284 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-14 07:06:47,284 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709, journal: Splitting hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709, size=1.1 K (1148bytes) at 1689318407233Finishing writing output for hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709 so closing down at 1689318407261 (+28 ms)Creating recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709.temp at 1689318407269 (+8 ms)3 split writer threads finished at 1689318407271 (+2 ms)Closed recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689318407282 (+11 ms)Rename recovered edits hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709.temp to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000013 at 1689318407284 (+2 ms)Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689318407284 2023-07-14 07:06:47,284 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709 2023-07-14 07:06:47,287 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-14 07:06:47,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] master.HMaster(3641): Remote procedure done, pid=14 2023-07-14 07:06:47,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] wal.WALSplitUtil(141): Moved hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting/jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709 to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs 2023-07-14 07:06:47,299 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-14 07:06:47,299 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709, worker=jenkins-hbase4.apache.org,33537,1689318403316 in 250 msec 2023-07-14 07:06:47,301 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:47,305 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-14 07:06:47,305 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C45707%2C1689318403359.1689318404709, worker=jenkins-hbase4.apache.org,33537,1689318403316 in 266 msec 2023-07-14 07:06:47,307 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting dir is empty, no logs to split. 2023-07-14 07:06:47,307 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,45707,1689318403359 WAL count=0, meta=false 2023-07-14 07:06:47,307 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,45707,1689318403359 WAL splitting is done? wals=0, meta=false 2023-07-14 07:06:47,310 INFO [PEWorker-4] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,45707,1689318403359 failed, ignore...File hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,45707,1689318403359-splitting does not exist. 2023-07-14 07:06:47,312 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN}] 2023-07-14 07:06:47,313 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN 2023-07-14 07:06:47,314 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 07:06:47,433 DEBUG [hconnection-0x1ef800db-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45707 this server is in the failed servers list 2023-07-14 07:06:47,465 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=10a151fabcc53ca1e27f811541e9729d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:47,465 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689318407465"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689318407465"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689318407465"}]},"ts":"1689318407465"} 2023-07-14 07:06:47,468 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 10a151fabcc53ca1e27f811541e9729d, server=jenkins-hbase4.apache.org,33537,1689318403316}] 2023-07-14 07:06:47,544 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 07:06:47,626 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:47,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 10a151fabcc53ca1e27f811541e9729d, NAME => 't,,1689318406051.10a151fabcc53ca1e27f811541e9729d.', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:06:47,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:47,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:06:47,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:47,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:47,629 INFO [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:47,630 DEBUG [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/f 2023-07-14 07:06:47,630 DEBUG [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/f 2023-07-14 07:06:47,631 INFO [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 10a151fabcc53ca1e27f811541e9729d columnFamilyName f 2023-07-14 07:06:47,631 INFO [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] regionserver.HStore(310): Store=10a151fabcc53ca1e27f811541e9729d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:06:47,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:47,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:47,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000013 2023-07-14 07:06:47,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000013 2023-07-14 07:06:47,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 10a151fabcc53ca1e27f811541e9729d 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-14 07:06:47,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/.tmp/f/8f07f06ba98f466499f6e8adecd3fb7b 2023-07-14 07:06:47,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/.tmp/f/8f07f06ba98f466499f6e8adecd3fb7b as hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/f/8f07f06ba98f466499f6e8adecd3fb7b 2023-07-14 07:06:47,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/f/8f07f06ba98f466499f6e8adecd3fb7b, entries=10, sequenceid=13, filesize=4.8 K 2023-07-14 07:06:47,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 10a151fabcc53ca1e27f811541e9729d in 112ms, sequenceid=13, compaction requested=false; wal=null 2023-07-14 07:06:47,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-14 07:06:47,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/0000000000000000013 2023-07-14 07:06:47,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:06:47,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-14 07:06:47,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 10a151fabcc53ca1e27f811541e9729d; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62070600, jitterRate=-0.07507598400115967}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 07:06:47,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 10a151fabcc53ca1e27f811541e9729d: 2023-07-14 07:06:47,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689318406051.10a151fabcc53ca1e27f811541e9729d., pid=16, masterSystemTime=1689318407621 2023-07-14 07:06:47,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:47,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:47,775 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=10a151fabcc53ca1e27f811541e9729d, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:47,775 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689318407774"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689318407774"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689318407774"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689318407774"}]},"ts":"1689318407774"} 2023-07-14 07:06:47,780 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-14 07:06:47,780 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 10a151fabcc53ca1e27f811541e9729d, server=jenkins-hbase4.apache.org,33537,1689318403316 in 309 msec 2023-07-14 07:06:47,782 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-14 07:06:47,782 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,45707,1689318403359 after splitting done 2023-07-14 07:06:47,782 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN in 468 msec 2023-07-14 07:06:47,783 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,45707,1689318403359 from processing; numProcessing=0 2023-07-14 07:06:47,785 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,45707,1689318403359, splitWal=true, meta=false in 764 msec 2023-07-14 07:06:47,955 DEBUG [Listener at localhost/46223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0700f054 to 127.0.0.1:57240 2023-07-14 07:06:47,956 DEBUG [Listener at localhost/46223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:06:47,956 INFO [Listener at localhost/46223] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-14 07:06:47,969 INFO [Listener at localhost/46223] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=271 (was 295), OpenFileDescriptor=567 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=102 (was 102), ProcessCount=171 (was 171), AvailableMemoryMB=7031 (was 7036) 2023-07-14 07:06:47,980 INFO [Listener at localhost/46223] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=271, OpenFileDescriptor=567, MaxFileDescriptor=60000, SystemLoadAverage=102, ProcessCount=171, AvailableMemoryMB=7030 2023-07-14 07:06:47,981 INFO [Listener at localhost/46223] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:45707 2023-07-14 07:06:47,993 INFO [Listener at localhost/46223] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-14 07:06:47,993 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:47,993 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:47,993 INFO [Listener at localhost/46223] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 07:06:47,993 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 07:06:47,994 INFO [Listener at localhost/46223] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 07:06:47,994 INFO [Listener at localhost/46223] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 07:06:47,995 INFO [Listener at localhost/46223] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37943 2023-07-14 07:06:47,996 INFO [Listener at localhost/46223] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 07:06:47,996 DEBUG [Listener at localhost/46223] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 07:06:47,997 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:47,998 INFO [Listener at localhost/46223] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 07:06:47,999 INFO [Listener at localhost/46223] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37943 connecting to ZooKeeper ensemble=127.0.0.1:57240 2023-07-14 07:06:48,003 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:379430x0, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 07:06:48,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:37943-0x10162a7a8040008 connected 2023-07-14 07:06:48,005 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(162): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 07:06:48,006 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(162): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-14 07:06:48,006 DEBUG [Listener at localhost/46223] zookeeper.ZKUtil(164): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 07:06:48,007 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37943 2023-07-14 07:06:48,007 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37943 2023-07-14 07:06:48,007 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37943 2023-07-14 07:06:48,008 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37943 2023-07-14 07:06:48,008 DEBUG [Listener at localhost/46223] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37943 2023-07-14 07:06:48,009 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(951): ClusterId : 2fc4a3dd-1b64-4a36-a4a9-8269119c0623 2023-07-14 07:06:48,010 DEBUG [RS:2;jenkins-hbase4:37943] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 07:06:48,013 DEBUG [RS:2;jenkins-hbase4:37943] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 07:06:48,013 DEBUG [RS:2;jenkins-hbase4:37943] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 07:06:48,016 DEBUG [RS:2;jenkins-hbase4:37943] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 07:06:48,017 DEBUG [RS:2;jenkins-hbase4:37943] zookeeper.ReadOnlyZKClient(139): Connect 0x6f8022bb to 127.0.0.1:57240 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 07:06:48,022 DEBUG [RS:2;jenkins-hbase4:37943] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6147c13b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 07:06:48,023 DEBUG [RS:2;jenkins-hbase4:37943] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@657849cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-14 07:06:48,032 DEBUG [RS:2;jenkins-hbase4:37943] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:37943 2023-07-14 07:06:48,032 INFO [RS:2;jenkins-hbase4:37943] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 07:06:48,032 INFO [RS:2;jenkins-hbase4:37943] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 07:06:48,032 DEBUG [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 07:06:48,033 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,41343,1689318402146 with isa=jenkins-hbase4.apache.org/172.31.14.131:37943, startcode=1689318407993 2023-07-14 07:06:48,033 DEBUG [RS:2;jenkins-hbase4:37943] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 07:06:48,037 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47011, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 07:06:48,038 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41343] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:06:48,038 DEBUG [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d 2023-07-14 07:06:48,038 DEBUG [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:35789 2023-07-14 07:06:48,038 DEBUG [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 07:06:48,040 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:06:48,040 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:06:48,041 DEBUG [RS:2;jenkins-hbase4:37943] zookeeper.ZKUtil(162): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:06:48,041 WARN [RS:2;jenkins-hbase4:37943] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 07:06:48,041 INFO [RS:2;jenkins-hbase4:37943] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 07:06:48,041 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37943,1689318407993] 2023-07-14 07:06:48,041 DEBUG [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1948): logDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:06:48,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:48,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:06:48,051 DEBUG [RS:2;jenkins-hbase4:37943] zookeeper.ZKUtil(162): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:06:48,051 DEBUG [RS:2;jenkins-hbase4:37943] zookeeper.ZKUtil(162): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:06:48,052 DEBUG [RS:2;jenkins-hbase4:37943] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 07:06:48,052 INFO [RS:2;jenkins-hbase4:37943] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 07:06:48,055 INFO [RS:2;jenkins-hbase4:37943] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 07:06:48,055 INFO [RS:2;jenkins-hbase4:37943] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 07:06:48,055 INFO [RS:2;jenkins-hbase4:37943] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:48,056 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 07:06:48,057 INFO [RS:2;jenkins-hbase4:37943] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:48,057 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:48,057 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:48,057 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:48,057 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:48,057 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:48,057 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-14 07:06:48,057 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:48,057 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:48,058 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:48,058 DEBUG [RS:2;jenkins-hbase4:37943] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-14 07:06:48,058 INFO [RS:2;jenkins-hbase4:37943] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:48,058 INFO [RS:2;jenkins-hbase4:37943] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:48,058 INFO [RS:2;jenkins-hbase4:37943] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:48,071 INFO [RS:2;jenkins-hbase4:37943] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 07:06:48,071 INFO [RS:2;jenkins-hbase4:37943] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37943,1689318407993-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 07:06:48,083 INFO [RS:2;jenkins-hbase4:37943] regionserver.Replication(203): jenkins-hbase4.apache.org,37943,1689318407993 started 2023-07-14 07:06:48,084 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,37943,1689318407993, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37943, sessionid=0x10162a7a8040008 2023-07-14 07:06:48,084 INFO [Listener at localhost/46223] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:37943,5,FailOnTimeoutGroup] 2023-07-14 07:06:48,084 DEBUG [RS:2;jenkins-hbase4:37943] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 07:06:48,084 DEBUG [RS:2;jenkins-hbase4:37943] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:06:48,084 INFO [Listener at localhost/46223] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:06:48,084 DEBUG [RS:2;jenkins-hbase4:37943] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37943,1689318407993' 2023-07-14 07:06:48,084 DEBUG [RS:2;jenkins-hbase4:37943] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 07:06:48,084 INFO [Listener at localhost/46223] client.TestScannerTimeout(177): START ************ test3686b 2023-07-14 07:06:48,085 DEBUG [RS:2;jenkins-hbase4:37943] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 07:06:48,085 DEBUG [RS:2;jenkins-hbase4:37943] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 07:06:48,085 DEBUG [RS:2;jenkins-hbase4:37943] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 07:06:48,085 DEBUG [RS:2;jenkins-hbase4:37943] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:06:48,085 DEBUG [RS:2;jenkins-hbase4:37943] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37943,1689318407993' 2023-07-14 07:06:48,085 DEBUG [RS:2;jenkins-hbase4:37943] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 07:06:48,086 DEBUG [RS:2;jenkins-hbase4:37943] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 07:06:48,086 DEBUG [RS:2;jenkins-hbase4:37943] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 07:06:48,086 INFO [RS:2;jenkins-hbase4:37943] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 07:06:48,086 INFO [RS:2;jenkins-hbase4:37943] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 07:06:48,087 DEBUG [Listener at localhost/46223] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 07:06:48,087 DEBUG [Listener at localhost/46223] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:06:48,088 DEBUG [Listener at localhost/46223] zookeeper.ReadOnlyZKClient(139): Connect 0x4075ebb6 to 127.0.0.1:57240 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 07:06:48,093 DEBUG [Listener at localhost/46223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e5d4ec0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 07:06:48,095 DEBUG [hconnection-0x78946a4f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 07:06:48,098 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42254, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 07:06:48,190 INFO [RS:2;jenkins-hbase4:37943] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37943%2C1689318407993, suffix=, logDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,37943,1689318407993, archiveDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs, maxLogs=32 2023-07-14 07:06:48,210 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK] 2023-07-14 07:06:48,210 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK] 2023-07-14 07:06:48,214 INFO [RS:2;jenkins-hbase4:37943] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,37943,1689318407993/jenkins-hbase4.apache.org%2C37943%2C1689318407993.1689318408191 2023-07-14 07:06:48,214 DEBUG [RS:2;jenkins-hbase4:37943] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK], DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK]] 2023-07-14 07:06:50,492 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-14 07:06:50,568 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-14 07:06:50,569 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-14 07:06:52,309 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 07:06:52,310 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-14 07:07:03,102 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease -9101669594447216621 expired clientIPAndPort=172.31.14.131:42254, userName=jenkins, regionInfo=t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:05,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33537] regionserver.RSRpcServices(3094): Client tried to access missing scanner -9101669594447216621 2023-07-14 07:07:05,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33537] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:42254 deadline: 1689318485104, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-9101669594447216621'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-14 07:07:05,106 DEBUG [Listener at localhost/46223] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-9101669594447216621'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-9101669594447216621'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-9101669594447216621'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-14 07:07:05,110 DEBUG [Listener at localhost/46223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4075ebb6 to 127.0.0.1:57240 2023-07-14 07:07:05,110 DEBUG [Listener at localhost/46223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:07:05,110 INFO [Listener at localhost/46223] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-14 07:07:05,125 INFO [Listener at localhost/46223] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=294 (was 271) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:37943 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:35789 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57240@0x4075ebb6-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57240@0x4075ebb6-SendThread(127.0.0.1:57240) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: jenkins-hbase4:37943Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_599986960_17 at /127.0.0.1:35522 [Receiving block BP-517806925-172.31.14.131-1689318399097:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57240@0x6f8022bb-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57240@0x4075ebb6 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1691545687.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57240@0x6f8022bb sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1691545687.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_599986960_17 at /127.0.0.1:40082 [Receiving block BP-517806925-172.31.14.131-1689318399097:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d-prefix:jenkins-hbase4.apache.org,37943,1689318407993 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-517806925-172.31.14.131-1689318399097:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-517806925-172.31.14.131-1689318399097:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57240@0x6f8022bb-SendThread(127.0.0.1:57240) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS:2;jenkins-hbase4:37943-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=37943 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=571 (was 567) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=79 (was 102), ProcessCount=171 (was 171), AvailableMemoryMB=6983 (was 7030) 2023-07-14 07:07:05,138 INFO [Listener at localhost/46223] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=294, OpenFileDescriptor=571, MaxFileDescriptor=60000, SystemLoadAverage=79, ProcessCount=171, AvailableMemoryMB=6982 2023-07-14 07:07:05,138 INFO [Listener at localhost/46223] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:45707 2023-07-14 07:07:05,138 INFO [Listener at localhost/46223] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,45707,1689318403359 2023-07-14 07:07:05,138 INFO [Listener at localhost/46223] client.TestScannerTimeout(104): START************ test2772 2023-07-14 07:07:05,142 DEBUG [Listener at localhost/46223] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 07:07:05,142 DEBUG [Listener at localhost/46223] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:05,143 DEBUG [Listener at localhost/46223] zookeeper.ReadOnlyZKClient(139): Connect 0x3a1416ac to 127.0.0.1:57240 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 07:07:05,147 DEBUG [Listener at localhost/46223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@496228a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 07:07:05,147 ERROR [Listener at localhost/46223] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,33537,1689318403316: die! ***** 2023-07-14 07:07:05,147 ERROR [Listener at localhost/46223] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-14 07:07:05,147 DEBUG [Listener at localhost/46223] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-14 07:07:05,148 DEBUG [Listener at localhost/46223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-14 07:07:05,153 DEBUG [Listener at localhost/46223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-14 07:07:05,155 DEBUG [Listener at localhost/46223] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-14 07:07:05,173 INFO [Listener at localhost/46223] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1135607808, "init": 513802240, "max": 2051014656, "used": 590121128 }, "NonHeapMemoryUsage": { "committed": 122249216, "init": 2555904, "max": -1, "used": 119793280 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11958, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 197, "ResponseSize_max": 679, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 199, "ResponseSize_median": 202, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 678, "ResponseSize_95th_percentile": 678, "ResponseSize_98th_percentile": 678, "ResponseSize_99th_percentile": 678, "ResponseSize_99.9th_percentile": 678, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16684 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:57240", "tag.serverName": "jenkins-hbase4.apache.org,33537,1689318403316", "tag.clusterId": "2fc4a3dd-1b64-4a36-a4a9-8269119c0623", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 1, "maxStoreFileCount": 1, "memStoreSize": 3306, "storeFileSize": 4926, "maxStoreFileAge": 16808, "minStoreFileAge": 16808, "avgStoreFileAge": 16808, "numReferenceFiles": 0, "regionServerStartTime": 1689318403316, "averageRegionSize": 2744, "storeFileIndexSize": 320, "staticIndexSize": 31, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 1863680, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 46, "totalRowActionRequestCount": 57, "readRequestCount": 43, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 10, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 22, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 10, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 270, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 21, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 10, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 21, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 270, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.6158478770280658, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.1625989103112326, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-14 07:07:05,175 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41343] master.MasterRpcServices(612): jenkins-hbase4.apache.org,33537,1689318403316 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,33537,1689318403316: die! ***** 2023-07-14 07:07:05,176 INFO [Listener at localhost/46223] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,33537,1689318403316' ***** 2023-07-14 07:07:05,178 INFO [Listener at localhost/46223] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-14 07:07:05,179 INFO [RS:0;jenkins-hbase4:33537] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 07:07:05,179 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 07:07:05,179 INFO [RS:0;jenkins-hbase4:33537] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-14 07:07:05,180 INFO [RS:0;jenkins-hbase4:33537] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-14 07:07:05,180 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(3305): Received CLOSE for 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:05,180 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(3305): Received CLOSE for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:07:05,180 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:07:05,180 DEBUG [RS:0;jenkins-hbase4:33537] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c63626 to 127.0.0.1:57240 2023-07-14 07:07:05,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7e77fde02849b028ee4ee8683f32fb6f, disabling compactions & flushes 2023-07-14 07:07:05,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:05,181 DEBUG [RS:0;jenkins-hbase4:33537] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:07:05,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:05,181 INFO [RS:0;jenkins-hbase4:33537] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 07:07:05,181 INFO [RS:0;jenkins-hbase4:33537] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 07:07:05,181 INFO [RS:0;jenkins-hbase4:33537] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 07:07:05,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. after waiting 0 ms 2023-07-14 07:07:05,181 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-14 07:07:05,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:05,181 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-14 07:07:05,182 DEBUG [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, 7e77fde02849b028ee4ee8683f32fb6f=hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f., 10a151fabcc53ca1e27f811541e9729d=t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.} 2023-07-14 07:07:05,182 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 07:07:05,182 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 07:07:05,182 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 07:07:05,182 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:05,182 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 07:07:05,182 DEBUG [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1504): Waiting on 10a151fabcc53ca1e27f811541e9729d, 1588230740, 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:05,182 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 07:07:05,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:05,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7e77fde02849b028ee4ee8683f32fb6f: 2023-07-14 07:07:05,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:05,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 10a151fabcc53ca1e27f811541e9729d, disabling compactions & flushes 2023-07-14 07:07:05,184 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 3228 in region hbase:meta,,1.1588230740 2023-07-14 07:07:05,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:05,184 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 07:07:05,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:05,186 DEBUG [hconnection-0x6ed71bca-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 07:07:05,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. after waiting 0 ms 2023-07-14 07:07:05,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:05,187 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 07:07:05,187 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 07:07:05,188 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-14 07:07:05,190 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55962, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 07:07:05,190 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33537] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33537,1689318403316 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 07:07:05,190 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33537] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:55962 deadline: 1689318485190, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33537,1689318403316 aborting 2023-07-14 07:07:05,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:05,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 10a151fabcc53ca1e27f811541e9729d: 2023-07-14 07:07:05,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:05,297 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33537] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33537,1689318403316 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 07:07:05,297 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33537] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:55962 deadline: 1689318485297, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33537,1689318403316 aborting 2023-07-14 07:07:05,331 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 07:07:05,383 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,33537,1689318403316; all regions closed. 2023-07-14 07:07:05,397 DEBUG [RS:0;jenkins-hbase4:33537] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:07:05,397 INFO [RS:0;jenkins-hbase4:33537] regionserver.LeaseManager(133): Closed leases 2023-07-14 07:07:05,397 INFO [RS:0;jenkins-hbase4:33537] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-14 07:07:05,398 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 07:07:05,398 INFO [RS:0;jenkins-hbase4:33537] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33537 2023-07-14 07:07:05,404 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:07:05,404 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:07:05,404 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:07:05,404 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 2023-07-14 07:07:05,405 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:07:05,406 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33537,1689318403316] 2023-07-14 07:07:05,406 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33537,1689318403316; numProcessing=1 2023-07-14 07:07:05,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:05,407 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,33537,1689318403316 znode expired, triggering replicatorRemoved event 2023-07-14 07:07:05,407 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33537,1689318403316 already deleted, retry=false 2023-07-14 07:07:05,408 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,33537,1689318403316 on jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:07:05,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:05,413 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,33537,1689318403316, splitWal=true, meta=true 2023-07-14 07:07:05,413 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase4.apache.org,33537,1689318403316 (carryingMeta=true) jenkins-hbase4.apache.org,33537,1689318403316/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1ecc6a20[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-14 07:07:05,414 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33537,1689318403316, splitWal=true, meta=true 2023-07-14 07:07:05,416 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33537,1689318403316, splitWal=true, meta=true, isMeta: true 2023-07-14 07:07:05,420 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting 2023-07-14 07:07:05,422 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,33537,1689318403316 WAL count=1, meta=true 2023-07-14 07:07:05,422 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta}] 2023-07-14 07:07:05,425 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:05,426 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta, worker=jenkins-hbase4.apache.org,37943,1689318407993}] 2023-07-14 07:07:05,500 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:33537 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33537 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 07:07:05,502 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:33537 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33537 2023-07-14 07:07:05,506 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 07:07:05,506 INFO [RS:0;jenkins-hbase4:33537] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,33537,1689318403316; zookeeper connection closed. 2023-07-14 07:07:05,506 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:33537-0x10162a7a8040001, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 07:07:05,507 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2c766269] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2c766269 2023-07-14 07:07:05,578 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:05,579 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 07:07:05,580 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46558, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 07:07:05,581 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37943] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-14 07:07:05,597 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta, size=4.0 K (4123bytes) 2023-07-14 07:07:05,598 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta 2023-07-14 07:07:05,598 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta after 0ms 2023-07-14 07:07:05,603 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta took 6ms 2023-07-14 07:07:05,607 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-14 07:07:05,610 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta so closing down 2023-07-14 07:07:05,610 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 07:07:05,613 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta.temp 2023-07-14 07:07:05,613 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 07:07:05,657 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-14 07:07:05,659 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta.temp to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 07:07:05,660 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 57 ms; skipped=1; WAL=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false 2023-07-14 07:07:05,660 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta, journal: Splitting hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta, size=4.0 K (4123bytes) at 1689318425598Finishing writing output for hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta so closing down at 1689318425610 (+12 ms)Creating recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta.temp at 1689318425613 (+3 ms)3 split writer threads finished at 1689318425613Closed recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689318425657 (+44 ms)Rename recovered edits hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta.temp to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689318425660 (+3 ms)Processed 13 edits across 1 Regions in 57 ms; skipped=1; WAL=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false at 1689318425660 2023-07-14 07:07:05,660 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta 2023-07-14 07:07:05,660 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-14 07:07:05,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] master.HMaster(3641): Remote procedure done, pid=19 2023-07-14 07:07:05,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] wal.WALSplitUtil(141): Moved hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs 2023-07-14 07:07:05,667 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-14 07:07:05,667 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta, worker=jenkins-hbase4.apache.org,37943,1689318407993 in 239 msec 2023-07-14 07:07:05,669 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:05,671 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-14 07:07:05,671 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C33537%2C1689318403316.meta.1689318404899.meta, worker=jenkins-hbase4.apache.org,37943,1689318407993 in 247 msec 2023-07-14 07:07:05,673 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting dir is empty, no logs to split. 2023-07-14 07:07:05,673 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,33537,1689318403316 WAL count=0, meta=true 2023-07-14 07:07:05,674 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,33537,1689318403316 WAL splitting is done? wals=0, meta=true 2023-07-14 07:07:05,675 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-14 07:07:05,676 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-14 07:07:05,677 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 07:07:05,803 DEBUG [hconnection-0x6ed71bca-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:33537 this server is in the failed servers list 2023-07-14 07:07:05,827 DEBUG [jenkins-hbase4:41343] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-14 07:07:05,827 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37943,1689318407993, state=OPENING 2023-07-14 07:07:05,829 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 07:07:05,829 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37943,1689318407993}] 2023-07-14 07:07:05,829 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 07:07:05,986 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-14 07:07:05,987 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 07:07:05,989 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37943%2C1689318407993.meta, suffix=.meta, logDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,37943,1689318407993, archiveDir=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs, maxLogs=32 2023-07-14 07:07:06,008 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK] 2023-07-14 07:07:06,009 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK] 2023-07-14 07:07:06,011 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,37943,1689318407993/jenkins-hbase4.apache.org%2C37943%2C1689318407993.meta.1689318425991.meta 2023-07-14 07:07:06,012 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41169,DS-7a838618-123f-4d6b-86d0-84d6291e9886,DISK], DatanodeInfoWithStorage[127.0.0.1:33815,DS-0c17373b-6185-47ab-b29d-cc71331fb632,DISK]] 2023-07-14 07:07:06,012 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:07:06,012 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-14 07:07:06,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-14 07:07:06,013 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-14 07:07:06,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-14 07:07:06,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:07:06,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-14 07:07:06,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-14 07:07:06,015 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 07:07:06,017 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/info 2023-07-14 07:07:06,017 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/info 2023-07-14 07:07:06,018 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 07:07:06,019 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:07:06,020 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 07:07:06,021 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/rep_barrier 2023-07-14 07:07:06,021 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/rep_barrier 2023-07-14 07:07:06,021 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 07:07:06,022 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:07:06,022 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 07:07:06,023 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/table 2023-07-14 07:07:06,023 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/table 2023-07-14 07:07:06,024 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 07:07:06,024 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:07:06,025 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740 2023-07-14 07:07:06,028 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740 2023-07-14 07:07:06,028 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 07:07:06,036 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 07:07:06,036 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.15 KB heapSize=6.27 KB 2023-07-14 07:07:06,056 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/.tmp/info/a2b0e613e67b4fd8b4286d40d1b7b911 2023-07-14 07:07:06,082 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/.tmp/table/9514b27cca8c4389bbea9e63e79e6714 2023-07-14 07:07:06,090 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/.tmp/info/a2b0e613e67b4fd8b4286d40d1b7b911 as hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/info/a2b0e613e67b4fd8b4286d40d1b7b911 2023-07-14 07:07:06,099 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/info/a2b0e613e67b4fd8b4286d40d1b7b911, entries=28, sequenceid=15, filesize=7.7 K 2023-07-14 07:07:06,100 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/.tmp/table/9514b27cca8c4389bbea9e63e79e6714 as hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/table/9514b27cca8c4389bbea9e63e79e6714 2023-07-14 07:07:06,108 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/table/9514b27cca8c4389bbea9e63e79e6714, entries=4, sequenceid=15, filesize=4.7 K 2023-07-14 07:07:06,108 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.15 KB/3228, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 72ms, sequenceid=15, compaction requested=false; wal=null 2023-07-14 07:07:06,109 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 07:07:06,112 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 07:07:06,114 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 07:07:06,116 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-14 07:07:06,117 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75306138, jitterRate=0.12214890122413635}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 07:07:06,117 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 07:07:06,118 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689318425981 2023-07-14 07:07:06,120 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-14 07:07:06,121 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-14 07:07:06,122 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37943,1689318407993, state=OPEN 2023-07-14 07:07:06,124 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 07:07:06,124 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 07:07:06,127 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-14 07:07:06,127 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37943,1689318407993 in 295 msec 2023-07-14 07:07:06,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-14 07:07:06,130 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,33537,1689318403316 had 3 regions 2023-07-14 07:07:06,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 452 msec 2023-07-14 07:07:06,131 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33537,1689318403316, splitWal=true, meta=true, isMeta: false 2023-07-14 07:07:06,133 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,33537,1689318403316 WAL count=1, meta=false 2023-07-14 07:07:06,133 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709}] 2023-07-14 07:07:06,135 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:06,136 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709, worker=jenkins-hbase4.apache.org,37943,1689318407993}] 2023-07-14 07:07:06,289 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37943] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-14 07:07:06,302 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709, size=899 (899bytes) 2023-07-14 07:07:06,302 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709 2023-07-14 07:07:06,303 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709 after 1ms 2023-07-14 07:07:06,306 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709 took 4ms 2023-07-14 07:07:06,309 DEBUG [hconnection-0x6ed71bca-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 07:07:06,309 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 7e77fde02849b028ee4ee8683f32fb6f: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-14 07:07:06,313 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46562, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 07:07:06,313 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 10a151fabcc53ca1e27f811541e9729d: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-14 07:07:06,313 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709 so closing down 2023-07-14 07:07:06,314 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 07:07:06,316 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709.temp 2023-07-14 07:07:06,317 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 07:07:06,322 DEBUG [hconnection-0x6ed71bca-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:33537 this server is in the failed servers list 2023-07-14 07:07:06,331 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-14 07:07:06,333 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709.temp to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000005 2023-07-14 07:07:06,333 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 27 ms; skipped=2; WAL=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709, size=899, length=899, corrupted=false, cancelled=false 2023-07-14 07:07:06,334 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709, journal: Splitting hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709, size=899 (899bytes) at 1689318426302Finishing writing output for hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709 so closing down at 1689318426314 (+12 ms)Creating recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709.temp at 1689318426316 (+2 ms)3 split writer threads finished at 1689318426317 (+1 ms)Closed recovered edits writer path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689318426332 (+15 ms)Rename recovered edits hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709.temp to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000005 at 1689318426333 (+1 ms)Processed 4 edits across 1 Regions in 27 ms; skipped=2; WAL=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709, size=899, length=899, corrupted=false, cancelled=false at 1689318426333 2023-07-14 07:07:06,334 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709 2023-07-14 07:07:06,334 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-14 07:07:06,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] master.HMaster(3641): Remote procedure done, pid=23 2023-07-14 07:07:06,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41343] wal.WALSplitUtil(141): Moved hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting/jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709 to hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs 2023-07-14 07:07:06,340 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-14 07:07:06,340 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709, worker=jenkins-hbase4.apache.org,37943,1689318407993 in 202 msec 2023-07-14 07:07:06,341 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:06,344 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-14 07:07:06,344 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C33537%2C1689318403316.1689318404709, worker=jenkins-hbase4.apache.org,37943,1689318407993 in 209 msec 2023-07-14 07:07:06,346 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/WALs/jenkins-hbase4.apache.org,33537,1689318403316-splitting dir is empty, no logs to split. 2023-07-14 07:07:06,346 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,33537,1689318403316 WAL count=0, meta=false 2023-07-14 07:07:06,346 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,33537,1689318403316 WAL splitting is done? wals=0, meta=false 2023-07-14 07:07:06,348 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7e77fde02849b028ee4ee8683f32fb6f, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN}] 2023-07-14 07:07:06,349 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7e77fde02849b028ee4ee8683f32fb6f, ASSIGN 2023-07-14 07:07:06,349 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN 2023-07-14 07:07:06,350 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7e77fde02849b028ee4ee8683f32fb6f, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 07:07:06,350 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 07:07:06,432 DEBUG [hconnection-0x6ed71bca-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:33537 this server is in the failed servers list 2023-07-14 07:07:06,500 DEBUG [jenkins-hbase4:41343] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-14 07:07:06,500 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=7e77fde02849b028ee4ee8683f32fb6f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:06,500 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=10a151fabcc53ca1e27f811541e9729d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:06,501 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689318426500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689318426500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689318426500"}]},"ts":"1689318426500"} 2023-07-14 07:07:06,501 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689318426500"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689318426500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689318426500"}]},"ts":"1689318426500"} 2023-07-14 07:07:06,501 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:33537 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33537 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 07:07:06,503 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:33537 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33537 2023-07-14 07:07:06,606 DEBUG [PEWorker-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 07:07:06,607 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 07:07:06,609 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; OpenRegionProcedure 10a151fabcc53ca1e27f811541e9729d, server=jenkins-hbase4.apache.org,37943,1689318407993}] 2023-07-14 07:07:06,611 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure 7e77fde02849b028ee4ee8683f32fb6f, server=jenkins-hbase4.apache.org,37943,1689318407993}] 2023-07-14 07:07:06,638 DEBUG [hconnection-0x6ed71bca-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:33537 this server is in the failed servers list 2023-07-14 07:07:06,766 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:06,766 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7e77fde02849b028ee4ee8683f32fb6f, NAME => 'hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:07:06,767 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:06,767 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:07:06,767 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:06,767 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:06,769 INFO [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:06,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:06,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 10a151fabcc53ca1e27f811541e9729d, NAME => 't,,1689318406051.10a151fabcc53ca1e27f811541e9729d.', STARTKEY => '', ENDKEY => ''} 2023-07-14 07:07:06,770 DEBUG [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/info 2023-07-14 07:07:06,770 DEBUG [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/info 2023-07-14 07:07:06,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:07:06,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 07:07:06,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:07:06,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:07:06,770 INFO [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7e77fde02849b028ee4ee8683f32fb6f columnFamilyName info 2023-07-14 07:07:06,771 INFO [StoreOpener-7e77fde02849b028ee4ee8683f32fb6f-1] regionserver.HStore(310): Store=7e77fde02849b028ee4ee8683f32fb6f/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:07:06,772 INFO [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:07:06,773 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:06,774 DEBUG [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/f 2023-07-14 07:07:06,774 DEBUG [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/f 2023-07-14 07:07:06,774 INFO [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 10a151fabcc53ca1e27f811541e9729d columnFamilyName f 2023-07-14 07:07:06,775 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:06,776 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000005 2023-07-14 07:07:06,780 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000005 2023-07-14 07:07:06,780 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7e77fde02849b028ee4ee8683f32fb6f 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-14 07:07:06,784 DEBUG [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] regionserver.HStore(539): loaded hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/f/8f07f06ba98f466499f6e8adecd3fb7b 2023-07-14 07:07:06,785 INFO [StoreOpener-10a151fabcc53ca1e27f811541e9729d-1] regionserver.HStore(310): Store=10a151fabcc53ca1e27f811541e9729d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 07:07:06,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:07:06,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:07:06,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:07:06,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-14 07:07:06,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 10a151fabcc53ca1e27f811541e9729d; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63601114, jitterRate=-0.052269548177719116}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 07:07:06,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 10a151fabcc53ca1e27f811541e9729d: 2023-07-14 07:07:06,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689318406051.10a151fabcc53ca1e27f811541e9729d., pid=26, masterSystemTime=1689318426762 2023-07-14 07:07:06,808 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/.tmp/info/07b4c5fb079d4d27b78e39142134cd80 2023-07-14 07:07:06,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:06,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:06,809 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=10a151fabcc53ca1e27f811541e9729d, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:06,809 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689318406051.10a151fabcc53ca1e27f811541e9729d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689318426809"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689318426809"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689318426809"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689318426809"}]},"ts":"1689318426809"} 2023-07-14 07:07:06,817 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-14 07:07:06,817 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; OpenRegionProcedure 10a151fabcc53ca1e27f811541e9729d, server=jenkins-hbase4.apache.org,37943,1689318407993 in 205 msec 2023-07-14 07:07:06,818 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=10a151fabcc53ca1e27f811541e9729d, ASSIGN in 469 msec 2023-07-14 07:07:06,819 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/.tmp/info/07b4c5fb079d4d27b78e39142134cd80 as hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/info/07b4c5fb079d4d27b78e39142134cd80 2023-07-14 07:07:06,826 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/info/07b4c5fb079d4d27b78e39142134cd80, entries=2, sequenceid=5, filesize=4.8 K 2023-07-14 07:07:06,826 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 7e77fde02849b028ee4ee8683f32fb6f in 46ms, sequenceid=5, compaction requested=false; wal=null 2023-07-14 07:07:06,828 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/0000000000000000005 2023-07-14 07:07:06,831 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:06,834 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-14 07:07:06,835 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7e77fde02849b028ee4ee8683f32fb6f; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69377188, jitterRate=0.03380066156387329}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 07:07:06,835 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7e77fde02849b028ee4ee8683f32fb6f: 2023-07-14 07:07:06,836 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f., pid=27, masterSystemTime=1689318426762 2023-07-14 07:07:06,838 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:06,838 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:06,839 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=7e77fde02849b028ee4ee8683f32fb6f, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:06,839 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689318426839"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689318426839"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689318426839"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689318426839"}]},"ts":"1689318426839"} 2023-07-14 07:07:06,843 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-07-14 07:07:06,844 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure 7e77fde02849b028ee4ee8683f32fb6f, server=jenkins-hbase4.apache.org,37943,1689318407993 in 231 msec 2023-07-14 07:07:06,846 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=17 2023-07-14 07:07:06,847 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,33537,1689318403316 after splitting done 2023-07-14 07:07:06,847 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7e77fde02849b028ee4ee8683f32fb6f, ASSIGN in 496 msec 2023-07-14 07:07:06,847 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,33537,1689318403316 from processing; numProcessing=0 2023-07-14 07:07:06,849 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,33537,1689318403316, splitWal=true, meta=true in 1.4390 sec 2023-07-14 07:07:06,954 DEBUG [Listener at localhost/46223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a1416ac to 127.0.0.1:57240 2023-07-14 07:07:06,954 DEBUG [Listener at localhost/46223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:07:06,955 INFO [Listener at localhost/46223] client.TestScannerTimeout(124): END ************ test2772 2023-07-14 07:07:06,982 INFO [Listener at localhost/46223] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=276 (was 294), OpenFileDescriptor=583 (was 571) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=81 (was 79) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=6909 (was 6982) 2023-07-14 07:07:06,982 INFO [Listener at localhost/46223] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-14 07:07:06,982 INFO [Listener at localhost/46223] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-14 07:07:06,983 DEBUG [Listener at localhost/46223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65b7854e to 127.0.0.1:57240 2023-07-14 07:07:06,984 DEBUG [Listener at localhost/46223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:07:06,984 DEBUG [Listener at localhost/46223] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-14 07:07:06,984 DEBUG [Listener at localhost/46223] util.JVMClusterUtil(257): Found active master hash=687717465, stopped=false 2023-07-14 07:07:06,984 INFO [Listener at localhost/46223] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:07:06,992 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-14 07:07:06,992 INFO [Listener at localhost/46223] procedure2.ProcedureExecutor(629): Stopping 2023-07-14 07:07:06,992 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-14 07:07:06,992 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:07:06,992 DEBUG [Listener at localhost/46223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d673313 to 127.0.0.1:57240 2023-07-14 07:07:06,992 DEBUG [Listener at localhost/46223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:07:06,992 INFO [Listener at localhost/46223] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,37943,1689318407993' ***** 2023-07-14 07:07:06,992 INFO [Listener at localhost/46223] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-14 07:07:06,993 INFO [RS:2;jenkins-hbase4:37943] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 07:07:06,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 07:07:06,993 INFO [RS:2;jenkins-hbase4:37943] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-14 07:07:06,993 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 07:07:06,993 INFO [RS:2;jenkins-hbase4:37943] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-14 07:07:06,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 07:07:06,993 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(3305): Received CLOSE for 10a151fabcc53ca1e27f811541e9729d 2023-07-14 07:07:06,994 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(3305): Received CLOSE for 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:06,994 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:06,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 10a151fabcc53ca1e27f811541e9729d, disabling compactions & flushes 2023-07-14 07:07:06,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:06,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:06,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. after waiting 0 ms 2023-07-14 07:07:06,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:06,995 DEBUG [RS:2;jenkins-hbase4:37943] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f8022bb to 127.0.0.1:57240 2023-07-14 07:07:06,995 DEBUG [RS:2;jenkins-hbase4:37943] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:07:06,995 INFO [RS:2;jenkins-hbase4:37943] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 07:07:06,995 INFO [RS:2;jenkins-hbase4:37943] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 07:07:06,995 INFO [RS:2;jenkins-hbase4:37943] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 07:07:06,995 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-14 07:07:06,999 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-14 07:07:07,000 DEBUG [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, 10a151fabcc53ca1e27f811541e9729d=t,,1689318406051.10a151fabcc53ca1e27f811541e9729d., 7e77fde02849b028ee4ee8683f32fb6f=hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f.} 2023-07-14 07:07:07,000 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 07:07:07,000 DEBUG [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1504): Waiting on 10a151fabcc53ca1e27f811541e9729d, 1588230740, 7e77fde02849b028ee4ee8683f32fb6f 2023-07-14 07:07:07,000 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 07:07:07,000 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 07:07:07,000 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 07:07:07,000 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 07:07:07,000 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-14 07:07:07,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/default/t/10a151fabcc53ca1e27f811541e9729d/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-14 07:07:07,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:07,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 10a151fabcc53ca1e27f811541e9729d: 2023-07-14 07:07:07,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689318406051.10a151fabcc53ca1e27f811541e9729d. 2023-07-14 07:07:07,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7e77fde02849b028ee4ee8683f32fb6f, disabling compactions & flushes 2023-07-14 07:07:07,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:07,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:07,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. after waiting 0 ms 2023-07-14 07:07:07,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:07,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/namespace/7e77fde02849b028ee4ee8683f32fb6f/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-14 07:07:07,014 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:07,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7e77fde02849b028ee4ee8683f32fb6f: 2023-07-14 07:07:07,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689318405168.7e77fde02849b028ee4ee8683f32fb6f. 2023-07-14 07:07:07,029 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/.tmp/info/3129d662e1494d83bce9659068d0ac27 2023-07-14 07:07:07,038 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/.tmp/info/3129d662e1494d83bce9659068d0ac27 as hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/info/3129d662e1494d83bce9659068d0ac27 2023-07-14 07:07:07,046 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/info/3129d662e1494d83bce9659068d0ac27, entries=16, sequenceid=22, filesize=6.4 K 2023-07-14 07:07:07,047 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 47ms, sequenceid=22, compaction requested=false 2023-07-14 07:07:07,055 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-14 07:07:07,056 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 07:07:07,057 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 07:07:07,057 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 07:07:07,057 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-14 07:07:07,060 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 07:07:07,136 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-14 07:07:07,136 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-14 07:07:07,200 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,37943,1689318407993; all regions closed. 2023-07-14 07:07:07,208 DEBUG [RS:2;jenkins-hbase4:37943] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs 2023-07-14 07:07:07,208 INFO [RS:2;jenkins-hbase4:37943] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37943%2C1689318407993.meta:.meta(num 1689318425991) 2023-07-14 07:07:07,214 DEBUG [RS:2;jenkins-hbase4:37943] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/oldWALs 2023-07-14 07:07:07,214 INFO [RS:2;jenkins-hbase4:37943] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37943%2C1689318407993:(num 1689318408191) 2023-07-14 07:07:07,214 DEBUG [RS:2;jenkins-hbase4:37943] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:07:07,214 INFO [RS:2;jenkins-hbase4:37943] regionserver.LeaseManager(133): Closed leases 2023-07-14 07:07:07,214 INFO [RS:2;jenkins-hbase4:37943] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-14 07:07:07,214 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 07:07:07,215 INFO [RS:2;jenkins-hbase4:37943] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37943 2023-07-14 07:07:07,217 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37943,1689318407993 2023-07-14 07:07:07,217 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:07:07,217 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 07:07:07,220 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37943,1689318407993] 2023-07-14 07:07:07,220 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37943,1689318407993; numProcessing=1 2023-07-14 07:07:07,221 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37943,1689318407993 already deleted, retry=false 2023-07-14 07:07:07,221 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37943,1689318407993 expired; onlineServers=0 2023-07-14 07:07:07,222 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,41343,1689318402146' ***** 2023-07-14 07:07:07,222 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-14 07:07:07,222 DEBUG [M:0;jenkins-hbase4:41343] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69c0b662, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-14 07:07:07,222 INFO [M:0;jenkins-hbase4:41343] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,41343,1689318402146 2023-07-14 07:07:07,222 INFO [M:0;jenkins-hbase4:41343] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,41343,1689318402146; all regions closed. 2023-07-14 07:07:07,222 DEBUG [M:0;jenkins-hbase4:41343] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 07:07:07,222 DEBUG [M:0;jenkins-hbase4:41343] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-14 07:07:07,223 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-14 07:07:07,223 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689318404389] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689318404389,5,FailOnTimeoutGroup] 2023-07-14 07:07:07,223 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689318404389] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689318404389,5,FailOnTimeoutGroup] 2023-07-14 07:07:07,223 DEBUG [M:0;jenkins-hbase4:41343] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-14 07:07:07,223 INFO [M:0;jenkins-hbase4:41343] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-14 07:07:07,223 INFO [M:0;jenkins-hbase4:41343] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-14 07:07:07,223 INFO [M:0;jenkins-hbase4:41343] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-14 07:07:07,224 DEBUG [M:0;jenkins-hbase4:41343] master.HMaster(1512): Stopping service threads 2023-07-14 07:07:07,224 INFO [M:0;jenkins-hbase4:41343] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-14 07:07:07,224 ERROR [M:0;jenkins-hbase4:41343] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1166221935) connection to localhost/127.0.0.1:35789 from jenkins,5,PEWorkerGroup] 2023-07-14 07:07:07,224 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-14 07:07:07,224 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 07:07:07,224 INFO [M:0;jenkins-hbase4:41343] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-14 07:07:07,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 07:07:07,225 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-14 07:07:07,225 DEBUG [M:0;jenkins-hbase4:41343] zookeeper.ZKUtil(398): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-14 07:07:07,225 WARN [M:0;jenkins-hbase4:41343] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-14 07:07:07,225 INFO [M:0;jenkins-hbase4:41343] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-14 07:07:07,225 INFO [M:0;jenkins-hbase4:41343] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-14 07:07:07,226 DEBUG [M:0;jenkins-hbase4:41343] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-14 07:07:07,226 INFO [M:0;jenkins-hbase4:41343] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 07:07:07,226 DEBUG [M:0;jenkins-hbase4:41343] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 07:07:07,226 DEBUG [M:0;jenkins-hbase4:41343] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-14 07:07:07,226 DEBUG [M:0;jenkins-hbase4:41343] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 07:07:07,226 INFO [M:0;jenkins-hbase4:41343] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=90.77 KB heapSize=109.49 KB 2023-07-14 07:07:07,242 INFO [M:0;jenkins-hbase4:41343] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=90.77 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8f69966fdfaa453481df273a930a2f38 2023-07-14 07:07:07,249 DEBUG [M:0;jenkins-hbase4:41343] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8f69966fdfaa453481df273a930a2f38 as hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8f69966fdfaa453481df273a930a2f38 2023-07-14 07:07:07,256 INFO [M:0;jenkins-hbase4:41343] regionserver.HStore(1080): Added hdfs://localhost:35789/user/jenkins/test-data/e5174fe6-e4b8-4079-67a4-535a18c7ca0d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8f69966fdfaa453481df273a930a2f38, entries=27, sequenceid=212, filesize=8.2 K 2023-07-14 07:07:07,257 INFO [M:0;jenkins-hbase4:41343] regionserver.HRegion(2948): Finished flush of dataSize ~90.77 KB/92953, heapSize ~109.48 KB/112104, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 31ms, sequenceid=212, compaction requested=false 2023-07-14 07:07:07,259 INFO [M:0;jenkins-hbase4:41343] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 07:07:07,259 DEBUG [M:0;jenkins-hbase4:41343] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 07:07:07,263 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 07:07:07,264 INFO [M:0;jenkins-hbase4:41343] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-14 07:07:07,264 INFO [M:0;jenkins-hbase4:41343] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41343 2023-07-14 07:07:07,266 DEBUG [M:0;jenkins-hbase4:41343] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,41343,1689318402146 already deleted, retry=false 2023-07-14 07:07:07,463 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 07:07:07,463 INFO [M:0;jenkins-hbase4:41343] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,41343,1689318402146; zookeeper connection closed. 2023-07-14 07:07:07,463 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): master:41343-0x10162a7a8040000, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 07:07:07,563 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 07:07:07,563 INFO [RS:2;jenkins-hbase4:37943] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,37943,1689318407993; zookeeper connection closed. 2023-07-14 07:07:07,563 DEBUG [Listener at localhost/46223-EventThread] zookeeper.ZKWatcher(600): regionserver:37943-0x10162a7a8040008, quorum=127.0.0.1:57240, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 07:07:07,569 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7e0b59b2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7e0b59b2 2023-07-14 07:07:07,569 INFO [Listener at localhost/46223] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-14 07:07:07,570 WARN [Listener at localhost/46223] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-14 07:07:07,582 INFO [Listener at localhost/46223] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 07:07:07,688 WARN [BP-517806925-172.31.14.131-1689318399097 heartbeating to localhost/127.0.0.1:35789] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-14 07:07:07,688 WARN [BP-517806925-172.31.14.131-1689318399097 heartbeating to localhost/127.0.0.1:35789] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-517806925-172.31.14.131-1689318399097 (Datanode Uuid 58e3e07d-dd68-4956-9014-3d9ed84cd780) service to localhost/127.0.0.1:35789 2023-07-14 07:07:07,690 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278/dfs/data/data3/current/BP-517806925-172.31.14.131-1689318399097] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 07:07:07,691 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278/dfs/data/data4/current/BP-517806925-172.31.14.131-1689318399097] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 07:07:07,692 WARN [Listener at localhost/46223] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-14 07:07:07,694 INFO [Listener at localhost/46223] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 07:07:07,799 WARN [BP-517806925-172.31.14.131-1689318399097 heartbeating to localhost/127.0.0.1:35789] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-14 07:07:07,799 WARN [BP-517806925-172.31.14.131-1689318399097 heartbeating to localhost/127.0.0.1:35789] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-517806925-172.31.14.131-1689318399097 (Datanode Uuid 0d61d807-3a42-401f-964f-0e3b86d434ea) service to localhost/127.0.0.1:35789 2023-07-14 07:07:07,800 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278/dfs/data/data1/current/BP-517806925-172.31.14.131-1689318399097] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 07:07:07,800 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/43b795b0-a268-7c22-7ba6-2ffae14a481b/cluster_060c148e-7de1-22fd-873b-affe4a5c7278/dfs/data/data2/current/BP-517806925-172.31.14.131-1689318399097] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 07:07:07,841 INFO [Listener at localhost/46223] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 07:07:07,954 INFO [Listener at localhost/46223] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-14 07:07:07,988 INFO [Listener at localhost/46223] hbase.HBaseTestingUtility(1293): Minicluster is down