2023-07-19 18:05:18,564 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-19 18:05:18,806 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80 2023-07-19 18:05:18,819 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-19 18:05:18,819 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe, deleteOnExit=true 2023-07-19 18:05:18,819 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-19 18:05:18,820 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/test.cache.data in system properties and HBase conf 2023-07-19 18:05:18,820 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/hadoop.tmp.dir in system properties and HBase conf 2023-07-19 18:05:18,821 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/hadoop.log.dir in system properties and HBase conf 2023-07-19 18:05:18,821 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-19 18:05:18,822 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-19 18:05:18,822 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-19 18:05:18,940 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-19 18:05:19,336 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-19 18:05:19,339 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-19 18:05:19,340 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-19 18:05:19,340 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-19 18:05:19,340 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-19 18:05:19,341 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-19 18:05:19,341 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-19 18:05:19,341 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-19 18:05:19,341 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-19 18:05:19,342 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-19 18:05:19,342 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/nfs.dump.dir in system properties and HBase conf 2023-07-19 18:05:19,342 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/java.io.tmpdir in system properties and HBase conf 2023-07-19 18:05:19,342 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-19 18:05:19,343 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-19 18:05:19,343 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-19 18:05:19,865 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-19 18:05:19,870 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-19 18:05:20,228 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-19 18:05:20,427 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-19 18:05:20,442 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 18:05:20,477 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 18:05:20,511 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/java.io.tmpdir/Jetty_localhost_46193_hdfs____.k900zx/webapp 2023-07-19 18:05:20,692 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46193 2023-07-19 18:05:20,703 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-19 18:05:20,704 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-19 18:05:21,122 WARN [Listener at localhost/46847] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 18:05:21,232 WARN [Listener at localhost/46847] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-19 18:05:21,250 WARN [Listener at localhost/46847] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 18:05:21,257 INFO [Listener at localhost/46847] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 18:05:21,263 INFO [Listener at localhost/46847] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/java.io.tmpdir/Jetty_localhost_36207_datanode____r0ze59/webapp 2023-07-19 18:05:21,363 INFO [Listener at localhost/46847] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36207 2023-07-19 18:05:21,711 WARN [Listener at localhost/42233] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 18:05:21,726 WARN [Listener at localhost/42233] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-19 18:05:21,729 WARN [Listener at localhost/42233] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 18:05:21,731 INFO [Listener at localhost/42233] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 18:05:21,739 INFO [Listener at localhost/42233] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/java.io.tmpdir/Jetty_localhost_44821_datanode____f6qwri/webapp 2023-07-19 18:05:21,836 INFO [Listener at localhost/42233] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44821 2023-07-19 18:05:21,845 WARN [Listener at localhost/37991] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 18:05:22,229 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf28e67c6c98e05fd: Processing first storage report for DS-448ed352-4365-4f01-ad82-b1ffa623ca8c from datanode 444984d7-9db9-44ec-a6e3-b3c00117edd9 2023-07-19 18:05:22,231 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf28e67c6c98e05fd: from storage DS-448ed352-4365-4f01-ad82-b1ffa623ca8c node DatanodeRegistration(127.0.0.1:38129, datanodeUuid=444984d7-9db9-44ec-a6e3-b3c00117edd9, infoPort=42039, infoSecurePort=0, ipcPort=42233, storageInfo=lv=-57;cid=testClusterID;nsid=1728226879;c=1689789919967), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-19 18:05:22,231 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe12f30c31eac93ef: Processing first storage report for DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe from datanode 2190df25-9c9e-4352-9e82-1a6fced49c9e 2023-07-19 18:05:22,232 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe12f30c31eac93ef: from storage DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe node DatanodeRegistration(127.0.0.1:45237, datanodeUuid=2190df25-9c9e-4352-9e82-1a6fced49c9e, infoPort=36269, infoSecurePort=0, ipcPort=37991, storageInfo=lv=-57;cid=testClusterID;nsid=1728226879;c=1689789919967), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-19 18:05:22,232 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf28e67c6c98e05fd: Processing first storage report for DS-9ae830ee-0432-4d76-b3de-4d10c1fdf85c from datanode 444984d7-9db9-44ec-a6e3-b3c00117edd9 2023-07-19 18:05:22,232 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf28e67c6c98e05fd: from storage DS-9ae830ee-0432-4d76-b3de-4d10c1fdf85c node DatanodeRegistration(127.0.0.1:38129, datanodeUuid=444984d7-9db9-44ec-a6e3-b3c00117edd9, infoPort=42039, infoSecurePort=0, ipcPort=42233, storageInfo=lv=-57;cid=testClusterID;nsid=1728226879;c=1689789919967), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 18:05:22,232 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe12f30c31eac93ef: Processing first storage report for DS-d8b8e683-ec00-4378-9148-3d4bbc24288a from datanode 2190df25-9c9e-4352-9e82-1a6fced49c9e 2023-07-19 18:05:22,232 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe12f30c31eac93ef: from storage DS-d8b8e683-ec00-4378-9148-3d4bbc24288a node DatanodeRegistration(127.0.0.1:45237, datanodeUuid=2190df25-9c9e-4352-9e82-1a6fced49c9e, infoPort=36269, infoSecurePort=0, ipcPort=37991, storageInfo=lv=-57;cid=testClusterID;nsid=1728226879;c=1689789919967), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 18:05:22,309 DEBUG [Listener at localhost/37991] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80 2023-07-19 18:05:22,383 INFO [Listener at localhost/37991] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe/zookeeper_0, clientPort=50575, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-19 18:05:22,398 INFO [Listener at localhost/37991] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50575 2023-07-19 18:05:22,406 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:22,408 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:23,100 INFO [Listener at localhost/37991] util.FSUtils(471): Created version file at hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1 with version=8 2023-07-19 18:05:23,100 INFO [Listener at localhost/37991] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/hbase-staging 2023-07-19 18:05:23,449 INFO [Listener at localhost/37991] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-19 18:05:23,943 INFO [Listener at localhost/37991] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 18:05:23,977 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:23,978 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:23,978 INFO [Listener at localhost/37991] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 18:05:23,978 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:23,978 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 18:05:24,136 INFO [Listener at localhost/37991] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 18:05:24,212 DEBUG [Listener at localhost/37991] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-19 18:05:24,313 INFO [Listener at localhost/37991] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42259 2023-07-19 18:05:24,324 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:24,327 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:24,348 INFO [Listener at localhost/37991] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42259 connecting to ZooKeeper ensemble=127.0.0.1:50575 2023-07-19 18:05:24,388 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:422590x0, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 18:05:24,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:42259-0x1017ec27f410000 connected 2023-07-19 18:05:24,426 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 18:05:24,427 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 18:05:24,432 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 18:05:24,443 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42259 2023-07-19 18:05:24,444 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42259 2023-07-19 18:05:24,446 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42259 2023-07-19 18:05:24,448 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42259 2023-07-19 18:05:24,448 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42259 2023-07-19 18:05:24,456 INFO [Listener at localhost/37991] master.HMaster(444): hbase.rootdir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1, hbase.cluster.distributed=false 2023-07-19 18:05:24,526 INFO [Listener at localhost/37991] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 18:05:24,526 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:24,526 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:24,527 INFO [Listener at localhost/37991] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 18:05:24,527 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:24,527 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 18:05:24,532 INFO [Listener at localhost/37991] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 18:05:24,535 INFO [Listener at localhost/37991] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44881 2023-07-19 18:05:24,537 INFO [Listener at localhost/37991] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 18:05:24,544 DEBUG [Listener at localhost/37991] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 18:05:24,546 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:24,548 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:24,550 INFO [Listener at localhost/37991] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44881 connecting to ZooKeeper ensemble=127.0.0.1:50575 2023-07-19 18:05:24,553 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:448810x0, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 18:05:24,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:44881-0x1017ec27f410001 connected 2023-07-19 18:05:24,555 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 18:05:24,556 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 18:05:24,558 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 18:05:24,558 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44881 2023-07-19 18:05:24,559 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44881 2023-07-19 18:05:24,560 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44881 2023-07-19 18:05:24,562 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44881 2023-07-19 18:05:24,562 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44881 2023-07-19 18:05:24,576 INFO [Listener at localhost/37991] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 18:05:24,576 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:24,576 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:24,577 INFO [Listener at localhost/37991] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 18:05:24,577 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:24,578 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 18:05:24,578 INFO [Listener at localhost/37991] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 18:05:24,580 INFO [Listener at localhost/37991] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42201 2023-07-19 18:05:24,580 INFO [Listener at localhost/37991] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 18:05:24,582 DEBUG [Listener at localhost/37991] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 18:05:24,583 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:24,585 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:24,587 INFO [Listener at localhost/37991] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42201 connecting to ZooKeeper ensemble=127.0.0.1:50575 2023-07-19 18:05:24,594 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:422010x0, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 18:05:24,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:42201-0x1017ec27f410002 connected 2023-07-19 18:05:24,595 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 18:05:24,596 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 18:05:24,597 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 18:05:24,598 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42201 2023-07-19 18:05:24,598 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42201 2023-07-19 18:05:24,603 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42201 2023-07-19 18:05:24,606 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42201 2023-07-19 18:05:24,610 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42201 2023-07-19 18:05:24,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:24,622 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-19 18:05:24,623 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:24,643 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 18:05:24,643 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 18:05:24,643 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 18:05:24,644 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:24,646 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 18:05:24,647 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,42259,1689789923265 from backup master directory 2023-07-19 18:05:24,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 18:05:24,650 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:24,650 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-19 18:05:24,653 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 18:05:24,653 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:24,656 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-19 18:05:24,679 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-19 18:05:24,771 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/hbase.id with ID: 675bcfc4-dd2a-40f5-ac0d-070575ea7225 2023-07-19 18:05:24,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:24,831 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:24,879 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x696f7b15 to 127.0.0.1:50575 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 18:05:24,914 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b2eeef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 18:05:24,938 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-19 18:05:24,941 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-19 18:05:24,961 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-19 18:05:24,961 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-19 18:05:24,963 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-19 18:05:24,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-19 18:05:24,968 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 18:05:25,002 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/data/master/store-tmp 2023-07-19 18:05:25,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:25,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-19 18:05:25,041 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 18:05:25,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 18:05:25,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-19 18:05:25,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 18:05:25,041 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 18:05:25,042 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 18:05:25,043 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/WALs/jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:25,071 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42259%2C1689789923265, suffix=, logDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/WALs/jenkins-hbase4.apache.org,42259,1689789923265, archiveDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/oldWALs, maxLogs=10 2023-07-19 18:05:25,153 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK] 2023-07-19 18:05:25,153 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK] 2023-07-19 18:05:25,162 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 18:05:25,267 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/WALs/jenkins-hbase4.apache.org,42259,1689789923265/jenkins-hbase4.apache.org%2C42259%2C1689789923265.1689789925086 2023-07-19 18:05:25,268 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK], DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK]] 2023-07-19 18:05:25,268 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:25,269 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:25,274 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 18:05:25,275 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 18:05:25,356 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-19 18:05:25,365 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-19 18:05:25,399 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-19 18:05:25,412 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:25,420 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-19 18:05:25,422 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-19 18:05:25,442 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 18:05:25,451 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 18:05:25,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69674288, jitterRate=0.03822779655456543}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 18:05:25,453 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 18:05:25,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-19 18:05:25,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-19 18:05:25,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-19 18:05:25,484 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-19 18:05:25,486 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-19 18:05:25,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 42 msec 2023-07-19 18:05:25,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-19 18:05:25,563 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-19 18:05:25,570 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-19 18:05:25,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-19 18:05:25,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-19 18:05:25,619 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-19 18:05:25,624 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-19 18:05:25,629 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-19 18:05:25,632 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:25,644 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-19 18:05:25,645 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-19 18:05:25,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-19 18:05:25,667 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 18:05:25,667 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 18:05:25,667 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 18:05:25,668 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:25,674 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,42259,1689789923265, sessionid=0x1017ec27f410000, setting cluster-up flag (Was=false) 2023-07-19 18:05:25,690 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:25,697 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-19 18:05:25,699 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:25,704 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:25,710 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-19 18:05:25,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:25,715 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.hbase-snapshot/.tmp 2023-07-19 18:05:25,815 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(951): ClusterId : 675bcfc4-dd2a-40f5-ac0d-070575ea7225 2023-07-19 18:05:25,822 INFO [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(951): ClusterId : 675bcfc4-dd2a-40f5-ac0d-070575ea7225 2023-07-19 18:05:25,826 DEBUG [RS:0;jenkins-hbase4:44881] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 18:05:25,826 DEBUG [RS:1;jenkins-hbase4:42201] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 18:05:25,835 DEBUG [RS:0;jenkins-hbase4:44881] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 18:05:25,835 DEBUG [RS:1;jenkins-hbase4:42201] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 18:05:25,835 DEBUG [RS:0;jenkins-hbase4:44881] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 18:05:25,835 DEBUG [RS:1;jenkins-hbase4:42201] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 18:05:25,840 DEBUG [RS:0;jenkins-hbase4:44881] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 18:05:25,840 DEBUG [RS:1;jenkins-hbase4:42201] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 18:05:25,841 DEBUG [RS:0;jenkins-hbase4:44881] zookeeper.ReadOnlyZKClient(139): Connect 0x3860d335 to 127.0.0.1:50575 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 18:05:25,842 DEBUG [RS:1;jenkins-hbase4:42201] zookeeper.ReadOnlyZKClient(139): Connect 0x4e1736b3 to 127.0.0.1:50575 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 18:05:25,850 DEBUG [RS:0;jenkins-hbase4:44881] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36a73a49, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 18:05:25,861 DEBUG [RS:0;jenkins-hbase4:44881] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3298b845, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 18:05:25,865 DEBUG [RS:1;jenkins-hbase4:42201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40d0d101, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 18:05:25,866 DEBUG [RS:1;jenkins-hbase4:42201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44ef5dc3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 18:05:25,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-19 18:05:25,887 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 18:05:25,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 18:05:25,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 18:05:25,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 18:05:25,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-19 18:05:25,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:25,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 18:05:25,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:25,891 DEBUG [RS:1;jenkins-hbase4:42201] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:42201 2023-07-19 18:05:25,891 DEBUG [RS:0;jenkins-hbase4:44881] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:44881 2023-07-19 18:05:25,899 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689789955899 2023-07-19 18:05:25,901 INFO [RS:0;jenkins-hbase4:44881] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 18:05:25,902 INFO [RS:0;jenkins-hbase4:44881] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 18:05:25,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-19 18:05:25,902 DEBUG [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 18:05:25,901 INFO [RS:1;jenkins-hbase4:42201] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 18:05:25,903 INFO [RS:1;jenkins-hbase4:42201] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 18:05:25,903 DEBUG [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 18:05:25,906 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42259,1689789923265 with isa=jenkins-hbase4.apache.org/172.31.14.131:42201, startcode=1689789924575 2023-07-19 18:05:25,906 INFO [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42259,1689789923265 with isa=jenkins-hbase4.apache.org/172.31.14.131:44881, startcode=1689789924525 2023-07-19 18:05:25,909 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-19 18:05:25,909 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-19 18:05:25,918 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-19 18:05:25,918 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-19 18:05:25,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-19 18:05:25,931 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-19 18:05:25,931 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-19 18:05:25,932 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-19 18:05:25,932 DEBUG [RS:0;jenkins-hbase4:44881] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 18:05:25,939 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:25,932 DEBUG [RS:1;jenkins-hbase4:42201] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 18:05:25,943 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-19 18:05:25,945 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-19 18:05:25,945 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-19 18:05:25,952 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-19 18:05:25,952 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-19 18:05:25,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689789925955,5,FailOnTimeoutGroup] 2023-07-19 18:05:25,963 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689789925960,5,FailOnTimeoutGroup] 2023-07-19 18:05:25,963 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:25,964 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-19 18:05:25,966 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:25,966 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,003 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-19 18:05:26,005 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-19 18:05:26,006 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1 2023-07-19 18:05:26,021 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39469, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 18:05:26,021 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49551, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 18:05:26,039 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42259] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:26,069 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42259] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:26,081 DEBUG [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1 2023-07-19 18:05:26,082 DEBUG [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46847 2023-07-19 18:05:26,082 DEBUG [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 18:05:26,083 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:26,084 DEBUG [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1 2023-07-19 18:05:26,084 DEBUG [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46847 2023-07-19 18:05:26,084 DEBUG [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 18:05:26,091 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 18:05:26,094 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:26,101 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/info 2023-07-19 18:05:26,102 DEBUG [RS:0;jenkins-hbase4:44881] zookeeper.ZKUtil(162): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:26,102 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 18:05:26,102 WARN [RS:0;jenkins-hbase4:44881] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 18:05:26,103 INFO [RS:0;jenkins-hbase4:44881] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 18:05:26,104 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42201,1689789924575] 2023-07-19 18:05:26,105 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44881,1689789924525] 2023-07-19 18:05:26,104 DEBUG [RS:1;jenkins-hbase4:42201] zookeeper.ZKUtil(162): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:26,104 DEBUG [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:26,105 WARN [RS:1;jenkins-hbase4:42201] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 18:05:26,105 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:26,105 INFO [RS:1;jenkins-hbase4:42201] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 18:05:26,106 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 18:05:26,106 DEBUG [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:26,110 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/rep_barrier 2023-07-19 18:05:26,114 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 18:05:26,116 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:26,116 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 18:05:26,119 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/table 2023-07-19 18:05:26,120 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 18:05:26,121 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:26,123 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740 2023-07-19 18:05:26,124 DEBUG [RS:0;jenkins-hbase4:44881] zookeeper.ZKUtil(162): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:26,124 DEBUG [RS:1;jenkins-hbase4:42201] zookeeper.ZKUtil(162): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:26,125 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740 2023-07-19 18:05:26,125 DEBUG [RS:1;jenkins-hbase4:42201] zookeeper.ZKUtil(162): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:26,126 DEBUG [RS:0;jenkins-hbase4:44881] zookeeper.ZKUtil(162): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:26,132 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 18:05:26,134 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 18:05:26,141 DEBUG [RS:0;jenkins-hbase4:44881] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 18:05:26,141 DEBUG [RS:1;jenkins-hbase4:42201] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 18:05:26,142 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 18:05:26,143 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65043487, jitterRate=-0.03077651560306549}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 18:05:26,143 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 18:05:26,143 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 18:05:26,144 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 18:05:26,144 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 18:05:26,144 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 18:05:26,144 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 18:05:26,147 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 18:05:26,148 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 18:05:26,154 INFO [RS:1;jenkins-hbase4:42201] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 18:05:26,155 INFO [RS:0;jenkins-hbase4:44881] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 18:05:26,157 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-19 18:05:26,157 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-19 18:05:26,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-19 18:05:26,180 INFO [RS:1;jenkins-hbase4:42201] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 18:05:26,186 INFO [RS:1;jenkins-hbase4:42201] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 18:05:26,187 INFO [RS:1;jenkins-hbase4:42201] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,188 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-19 18:05:26,188 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 18:05:26,182 INFO [RS:0;jenkins-hbase4:44881] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 18:05:26,189 INFO [RS:0;jenkins-hbase4:44881] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 18:05:26,190 INFO [RS:0;jenkins-hbase4:44881] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,192 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-19 18:05:26,193 INFO [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 18:05:26,198 INFO [RS:0;jenkins-hbase4:44881] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,198 INFO [RS:1;jenkins-hbase4:42201] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,199 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,199 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,199 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 18:05:26,200 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 18:05:26,200 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,200 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,201 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,201 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,201 DEBUG [RS:0;jenkins-hbase4:44881] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,201 DEBUG [RS:1;jenkins-hbase4:42201] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:26,204 INFO [RS:0;jenkins-hbase4:44881] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,205 INFO [RS:0;jenkins-hbase4:44881] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,205 INFO [RS:0;jenkins-hbase4:44881] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,206 INFO [RS:1;jenkins-hbase4:42201] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,206 INFO [RS:1;jenkins-hbase4:42201] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,207 INFO [RS:1;jenkins-hbase4:42201] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,223 INFO [RS:1;jenkins-hbase4:42201] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 18:05:26,223 INFO [RS:0;jenkins-hbase4:44881] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 18:05:26,227 INFO [RS:1;jenkins-hbase4:42201] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42201,1689789924575-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,227 INFO [RS:0;jenkins-hbase4:44881] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44881,1689789924525-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,251 INFO [RS:1;jenkins-hbase4:42201] regionserver.Replication(203): jenkins-hbase4.apache.org,42201,1689789924575 started 2023-07-19 18:05:26,251 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,42201,1689789924575, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42201, sessionid=0x1017ec27f410002 2023-07-19 18:05:26,254 INFO [RS:0;jenkins-hbase4:44881] regionserver.Replication(203): jenkins-hbase4.apache.org,44881,1689789924525 started 2023-07-19 18:05:26,254 DEBUG [RS:1;jenkins-hbase4:42201] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 18:05:26,255 INFO [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,44881,1689789924525, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44881, sessionid=0x1017ec27f410001 2023-07-19 18:05:26,255 DEBUG [RS:1;jenkins-hbase4:42201] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:26,255 DEBUG [RS:0;jenkins-hbase4:44881] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 18:05:26,255 DEBUG [RS:1;jenkins-hbase4:42201] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42201,1689789924575' 2023-07-19 18:05:26,255 DEBUG [RS:0;jenkins-hbase4:44881] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:26,256 DEBUG [RS:0;jenkins-hbase4:44881] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44881,1689789924525' 2023-07-19 18:05:26,256 DEBUG [RS:0;jenkins-hbase4:44881] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 18:05:26,256 DEBUG [RS:1;jenkins-hbase4:42201] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 18:05:26,260 DEBUG [RS:0;jenkins-hbase4:44881] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 18:05:26,260 DEBUG [RS:1;jenkins-hbase4:42201] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 18:05:26,260 DEBUG [RS:0;jenkins-hbase4:44881] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 18:05:26,260 DEBUG [RS:1;jenkins-hbase4:42201] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 18:05:26,261 DEBUG [RS:0;jenkins-hbase4:44881] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 18:05:26,261 DEBUG [RS:1;jenkins-hbase4:42201] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 18:05:26,261 DEBUG [RS:0;jenkins-hbase4:44881] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:26,261 DEBUG [RS:1;jenkins-hbase4:42201] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:26,262 DEBUG [RS:0;jenkins-hbase4:44881] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44881,1689789924525' 2023-07-19 18:05:26,262 DEBUG [RS:1;jenkins-hbase4:42201] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42201,1689789924575' 2023-07-19 18:05:26,262 DEBUG [RS:0;jenkins-hbase4:44881] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 18:05:26,262 DEBUG [RS:1;jenkins-hbase4:42201] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 18:05:26,263 DEBUG [RS:1;jenkins-hbase4:42201] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 18:05:26,263 DEBUG [RS:0;jenkins-hbase4:44881] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 18:05:26,264 DEBUG [RS:1;jenkins-hbase4:42201] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 18:05:26,264 DEBUG [RS:0;jenkins-hbase4:44881] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 18:05:26,264 INFO [RS:1;jenkins-hbase4:42201] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 18:05:26,264 INFO [RS:0;jenkins-hbase4:44881] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 18:05:26,265 INFO [RS:1;jenkins-hbase4:42201] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 18:05:26,265 INFO [RS:0;jenkins-hbase4:44881] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 18:05:26,345 DEBUG [jenkins-hbase4:42259] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-19 18:05:26,349 DEBUG [jenkins-hbase4:42259] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 18:05:26,360 DEBUG [jenkins-hbase4:42259] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 18:05:26,360 DEBUG [jenkins-hbase4:42259] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 18:05:26,360 DEBUG [jenkins-hbase4:42259] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 18:05:26,363 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42201,1689789924575, state=OPENING 2023-07-19 18:05:26,372 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-19 18:05:26,374 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:26,378 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 18:05:26,379 INFO [RS:0;jenkins-hbase4:44881] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44881%2C1689789924525, suffix=, logDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525, archiveDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs, maxLogs=32 2023-07-19 18:05:26,380 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42201,1689789924575}] 2023-07-19 18:05:26,381 INFO [RS:1;jenkins-hbase4:42201] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42201%2C1689789924575, suffix=, logDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575, archiveDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs, maxLogs=32 2023-07-19 18:05:26,412 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK] 2023-07-19 18:05:26,412 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK] 2023-07-19 18:05:26,412 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK] 2023-07-19 18:05:26,413 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK] 2023-07-19 18:05:26,425 INFO [RS:0;jenkins-hbase4:44881] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384 2023-07-19 18:05:26,425 INFO [RS:1;jenkins-hbase4:42201] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384 2023-07-19 18:05:26,426 DEBUG [RS:0;jenkins-hbase4:44881] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK], DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK]] 2023-07-19 18:05:26,426 DEBUG [RS:1;jenkins-hbase4:42201] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK], DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK]] 2023-07-19 18:05:26,572 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:26,575 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 18:05:26,579 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 18:05:26,591 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-19 18:05:26,592 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 18:05:26,596 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta, suffix=.meta, logDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575, archiveDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs, maxLogs=32 2023-07-19 18:05:26,616 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK] 2023-07-19 18:05:26,617 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK] 2023-07-19 18:05:26,624 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta 2023-07-19 18:05:26,625 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK], DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK]] 2023-07-19 18:05:26,625 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:26,627 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-19 18:05:26,649 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-19 18:05:26,687 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-19 18:05:26,692 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-19 18:05:26,693 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:26,693 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-19 18:05:26,693 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-19 18:05:26,696 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 18:05:26,698 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/info 2023-07-19 18:05:26,698 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/info 2023-07-19 18:05:26,699 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 18:05:26,700 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:26,701 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 18:05:26,703 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/rep_barrier 2023-07-19 18:05:26,703 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/rep_barrier 2023-07-19 18:05:26,704 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 18:05:26,705 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:26,705 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 18:05:26,707 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/table 2023-07-19 18:05:26,707 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/table 2023-07-19 18:05:26,708 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 18:05:26,709 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:26,711 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740 2023-07-19 18:05:26,715 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740 2023-07-19 18:05:26,718 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 18:05:26,721 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 18:05:26,723 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68280459, jitterRate=0.017458125948905945}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 18:05:26,723 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 18:05:26,734 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689789926564 2023-07-19 18:05:26,761 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-19 18:05:26,762 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42201,1689789924575, state=OPEN 2023-07-19 18:05:26,764 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-19 18:05:26,767 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 18:05:26,768 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 18:05:26,773 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-19 18:05:26,773 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42201,1689789924575 in 389 msec 2023-07-19 18:05:26,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-19 18:05:26,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 609 msec 2023-07-19 18:05:26,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.0050 sec 2023-07-19 18:05:26,790 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689789926790, completionTime=-1 2023-07-19 18:05:26,790 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-19 18:05:26,791 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-19 18:05:26,875 DEBUG [hconnection-0x76d641b4-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 18:05:26,879 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43794, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 18:05:26,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-19 18:05:26,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689789986896 2023-07-19 18:05:26,897 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689790046897 2023-07-19 18:05:26,897 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 105 msec 2023-07-19 18:05:26,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42259,1689789923265-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42259,1689789923265-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42259,1689789923265-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:42259, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:26,937 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-19 18:05:26,946 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-19 18:05:26,947 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-19 18:05:26,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-19 18:05:26,961 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-19 18:05:26,964 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-19 18:05:26,992 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:26,995 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75 empty. 2023-07-19 18:05:26,996 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:26,996 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-19 18:05:27,026 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-19 18:05:27,028 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => b172f58552c421ef659ce60b0365ef75, NAME => 'hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp 2023-07-19 18:05:27,050 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:27,050 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing b172f58552c421ef659ce60b0365ef75, disabling compactions & flushes 2023-07-19 18:05:27,051 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:27,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:27,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. after waiting 0 ms 2023-07-19 18:05:27,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:27,051 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:27,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for b172f58552c421ef659ce60b0365ef75: 2023-07-19 18:05:27,056 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-19 18:05:27,075 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689789927061"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689789927061"}]},"ts":"1689789927061"} 2023-07-19 18:05:27,105 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-19 18:05:27,107 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-19 18:05:27,112 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689789927107"}]},"ts":"1689789927107"} 2023-07-19 18:05:27,117 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-19 18:05:27,122 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 18:05:27,124 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 18:05:27,124 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 18:05:27,124 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 18:05:27,127 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b172f58552c421ef659ce60b0365ef75, ASSIGN}] 2023-07-19 18:05:27,130 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b172f58552c421ef659ce60b0365ef75, ASSIGN 2023-07-19 18:05:27,131 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=b172f58552c421ef659ce60b0365ef75, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42201,1689789924575; forceNewPlan=false, retain=false 2023-07-19 18:05:27,284 INFO [jenkins-hbase4:42259] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-19 18:05:27,285 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b172f58552c421ef659ce60b0365ef75, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:27,285 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689789927284"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689789927284"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689789927284"}]},"ts":"1689789927284"} 2023-07-19 18:05:27,289 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure b172f58552c421ef659ce60b0365ef75, server=jenkins-hbase4.apache.org,42201,1689789924575}] 2023-07-19 18:05:27,450 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:27,450 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b172f58552c421ef659ce60b0365ef75, NAME => 'hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:27,451 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:27,452 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:27,452 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:27,452 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:27,454 INFO [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:27,457 DEBUG [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/info 2023-07-19 18:05:27,457 DEBUG [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/info 2023-07-19 18:05:27,457 INFO [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b172f58552c421ef659ce60b0365ef75 columnFamilyName info 2023-07-19 18:05:27,458 INFO [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] regionserver.HStore(310): Store=b172f58552c421ef659ce60b0365ef75/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:27,460 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:27,461 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:27,465 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:27,469 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 18:05:27,470 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b172f58552c421ef659ce60b0365ef75; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60252915, jitterRate=-0.10216160118579865}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 18:05:27,470 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b172f58552c421ef659ce60b0365ef75: 2023-07-19 18:05:27,473 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75., pid=6, masterSystemTime=1689789927443 2023-07-19 18:05:27,479 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:27,479 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:27,480 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b172f58552c421ef659ce60b0365ef75, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:27,481 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689789927480"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689789927480"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689789927480"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689789927480"}]},"ts":"1689789927480"} 2023-07-19 18:05:27,491 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-19 18:05:27,491 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure b172f58552c421ef659ce60b0365ef75, server=jenkins-hbase4.apache.org,42201,1689789924575 in 198 msec 2023-07-19 18:05:27,495 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-19 18:05:27,496 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=b172f58552c421ef659ce60b0365ef75, ASSIGN in 364 msec 2023-07-19 18:05:27,497 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-19 18:05:27,497 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689789927497"}]},"ts":"1689789927497"} 2023-07-19 18:05:27,500 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-19 18:05:27,505 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-19 18:05:27,508 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 556 msec 2023-07-19 18:05:27,561 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-19 18:05:27,562 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-19 18:05:27,563 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:27,608 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-19 18:05:27,625 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-19 18:05:27,634 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 38 msec 2023-07-19 18:05:27,642 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-19 18:05:27,654 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-19 18:05:27,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-07-19 18:05:27,669 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-19 18:05:27,672 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-19 18:05:27,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.019sec 2023-07-19 18:05:27,675 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-19 18:05:27,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-19 18:05:27,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-19 18:05:27,678 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42259,1689789923265-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-19 18:05:27,679 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42259,1689789923265-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-19 18:05:27,687 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-19 18:05:27,725 DEBUG [Listener at localhost/37991] zookeeper.ReadOnlyZKClient(139): Connect 0x4789b34a to 127.0.0.1:50575 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 18:05:27,730 DEBUG [Listener at localhost/37991] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44a9743, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 18:05:27,751 DEBUG [hconnection-0x4e56a287-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 18:05:27,764 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 18:05:27,778 INFO [Listener at localhost/37991] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:27,792 DEBUG [Listener at localhost/37991] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-19 18:05:27,795 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-19 18:05:27,813 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-19 18:05:27,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-19 18:05:27,818 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-19 18:05:27,820 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-19 18:05:27,823 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-19 18:05:27,824 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:27,825 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db empty. 2023-07-19 18:05:27,827 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:27,827 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-19 18:05:27,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 18:05:27,890 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-19 18:05:27,892 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0ef15a5a788e7e8481c6f8908c6ae5db, NAME => 't,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/.tmp 2023-07-19 18:05:27,908 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:27,908 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 0ef15a5a788e7e8481c6f8908c6ae5db, disabling compactions & flushes 2023-07-19 18:05:27,908 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:27,908 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:27,908 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. after waiting 0 ms 2023-07-19 18:05:27,908 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:27,908 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:27,908 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 0ef15a5a788e7e8481c6f8908c6ae5db: 2023-07-19 18:05:27,913 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-19 18:05:27,915 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689789927915"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689789927915"}]},"ts":"1689789927915"} 2023-07-19 18:05:27,918 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-19 18:05:27,919 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-19 18:05:27,919 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689789927919"}]},"ts":"1689789927919"} 2023-07-19 18:05:27,922 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-19 18:05:27,926 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 18:05:27,927 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 18:05:27,927 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 18:05:27,927 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 18:05:27,928 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN}] 2023-07-19 18:05:27,930 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN 2023-07-19 18:05:27,932 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44881,1689789924525; forceNewPlan=false, retain=false 2023-07-19 18:05:27,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 18:05:28,082 INFO [jenkins-hbase4:42259] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-19 18:05:28,083 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0ef15a5a788e7e8481c6f8908c6ae5db, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:28,083 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689789928083"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689789928083"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689789928083"}]},"ts":"1689789928083"} 2023-07-19 18:05:28,086 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 0ef15a5a788e7e8481c6f8908c6ae5db, server=jenkins-hbase4.apache.org,44881,1689789924525}] 2023-07-19 18:05:28,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 18:05:28,245 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:28,246 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 18:05:28,255 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59242, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 18:05:28,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0ef15a5a788e7e8481c6f8908c6ae5db, NAME => 't,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:28,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:28,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:28,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:28,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:28,273 INFO [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:28,275 DEBUG [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/f 2023-07-19 18:05:28,275 DEBUG [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/f 2023-07-19 18:05:28,276 INFO [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0ef15a5a788e7e8481c6f8908c6ae5db columnFamilyName f 2023-07-19 18:05:28,276 INFO [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] regionserver.HStore(310): Store=0ef15a5a788e7e8481c6f8908c6ae5db/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:28,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:28,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:28,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:28,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 18:05:28,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0ef15a5a788e7e8481c6f8908c6ae5db; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74720286, jitterRate=0.11341902613639832}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 18:05:28,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0ef15a5a788e7e8481c6f8908c6ae5db: 2023-07-19 18:05:28,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db., pid=11, masterSystemTime=1689789928245 2023-07-19 18:05:28,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,299 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0ef15a5a788e7e8481c6f8908c6ae5db, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:28,299 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689789928299"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689789928299"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689789928299"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689789928299"}]},"ts":"1689789928299"} 2023-07-19 18:05:28,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-19 18:05:28,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 0ef15a5a788e7e8481c6f8908c6ae5db, server=jenkins-hbase4.apache.org,44881,1689789924525 in 217 msec 2023-07-19 18:05:28,309 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-19 18:05:28,309 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN in 379 msec 2023-07-19 18:05:28,310 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-19 18:05:28,310 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689789928310"}]},"ts":"1689789928310"} 2023-07-19 18:05:28,313 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-19 18:05:28,316 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-19 18:05:28,319 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 503 msec 2023-07-19 18:05:28,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 18:05:28,444 INFO [Listener at localhost/37991] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-19 18:05:28,444 DEBUG [Listener at localhost/37991] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-19 18:05:28,446 INFO [Listener at localhost/37991] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-19 18:05:28,449 INFO [Listener at localhost/37991] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-19 18:05:28,450 INFO [Listener at localhost/37991] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-19 18:05:28,450 INFO [Listener at localhost/37991] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-19 18:05:28,453 DEBUG [Listener at localhost/37991] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 18:05:28,457 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59252, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 18:05:28,509 INFO [Listener at localhost/37991] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=175, ProcessCount=171, AvailableMemoryMB=6499 2023-07-19 18:05:28,511 INFO [Listener at localhost/37991] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-19 18:05:28,514 DEBUG [Listener at localhost/37991] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 18:05:28,515 DEBUG [Listener at localhost/37991] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,515 INFO [Listener at localhost/37991] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-19 18:05:28,515 INFO [Listener at localhost/37991] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-19 18:05:28,519 INFO [Listener at localhost/37991] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689789927497/Put/vlen=2/seqid=0} 2023-07-19 18:05:28,519 INFO [Listener at localhost/37991] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-19 18:05:28,520 INFO [Listener at localhost/37991] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75./info:regioninfo/1689789927480/Put/vlen=41/seqid=0, hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75./info:seqnumDuringOpen/1689789927480/Put/vlen=8/seqid=0, hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75./info:server/1689789927480/Put/vlen=31/seqid=0, hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75./info:serverstartcode/1689789927480/Put/vlen=8/seqid=0, hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75./info:sn/1689789927284/Put/vlen=45/seqid=0, hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75./info:state/1689789927480/Put/vlen=4/seqid=0} 2023-07-19 18:05:28,520 INFO [Listener at localhost/37991] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => b172f58552c421ef659ce60b0365ef75, NAME => 'hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:28,520 INFO [Listener at localhost/37991] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689789928310/Put/vlen=2/seqid=0} 2023-07-19 18:05:28,520 INFO [Listener at localhost/37991] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-19 18:05:28,520 INFO [Listener at localhost/37991] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db./info:regioninfo/1689789928299/Put/vlen=35/seqid=0, t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db./info:seqnumDuringOpen/1689789928299/Put/vlen=8/seqid=0, t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db./info:server/1689789928299/Put/vlen=31/seqid=0, t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db./info:serverstartcode/1689789928299/Put/vlen=8/seqid=0, t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db./info:sn/1689789928083/Put/vlen=45/seqid=0, t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db./info:state/1689789928299/Put/vlen=4/seqid=0} 2023-07-19 18:05:28,521 INFO [Listener at localhost/37991] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 0ef15a5a788e7e8481c6f8908c6ae5db, NAME => 't,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:28,521 DEBUG [Listener at localhost/37991] zookeeper.ReadOnlyZKClient(139): Connect 0x7c11e80b to 127.0.0.1:50575 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 18:05:28,528 DEBUG [Listener at localhost/37991] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e8cbdf7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 18:05:28,528 INFO [Listener at localhost/37991] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-19 18:05:28,528 INFO [Listener at localhost/37991] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-19 18:05:28,531 DEBUG [hconnection-0x420100ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 18:05:28,534 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43818, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 18:05:28,538 DEBUG [hconnection-0x420100ae-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 18:05:28,541 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59268, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 18:05:28,543 INFO [Listener at localhost/37991] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-19 18:05:28,546 ERROR [Listener at localhost/37991] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,44881,1689789924525: die! ***** 2023-07-19 18:05:28,546 ERROR [Listener at localhost/37991] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-19 18:05:28,549 DEBUG [Listener at localhost/37991] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-19 18:05:28,552 DEBUG [Listener at localhost/37991] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-19 18:05:28,552 DEBUG [Listener at localhost/37991] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-19 18:05:28,553 DEBUG [Listener at localhost/37991] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-19 18:05:28,553 INFO [Listener at localhost/37991] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 973078528, "init": 513802240, "max": 2051014656, "used": 164936624 }, "NonHeapMemoryUsage": { "committed": 113205248, "init": 2555904, "max": -1, "used": 111176208 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-19 18:05:28,557 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42259] master.MasterRpcServices(612): jenkins-hbase4.apache.org,44881,1689789924525 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,44881,1689789924525: die! ***** 2023-07-19 18:05:28,559 INFO [Listener at localhost/37991] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,44881,1689789924525' ***** 2023-07-19 18:05:28,560 INFO [Listener at localhost/37991] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-19 18:05:28,560 INFO [RS:0;jenkins-hbase4:44881] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 18:05:28,560 INFO [RS:0;jenkins-hbase4:44881] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-19 18:05:28,560 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 18:05:28,560 INFO [RS:0;jenkins-hbase4:44881] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-19 18:05:28,561 INFO [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(3305): Received CLOSE for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:28,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44881] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 4467589083962540033 2023-07-19 18:05:28,561 INFO [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:28,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44881] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44881,1689789924525 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 18:05:28,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0ef15a5a788e7e8481c6f8908c6ae5db, disabling compactions & flushes 2023-07-19 18:05:28,562 DEBUG [RS:0;jenkins-hbase4:44881] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3860d335 to 127.0.0.1:50575 2023-07-19 18:05:28,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44881] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:59268 deadline: 1689789988561, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44881,1689789924525 aborting 2023-07-19 18:05:28,562 DEBUG [RS:0;jenkins-hbase4:44881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:28,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. after waiting 0 ms 2023-07-19 18:05:28,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,562 INFO [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-19 18:05:28,563 DEBUG [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1478): Online Regions={0ef15a5a788e7e8481c6f8908c6ae5db=t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.} 2023-07-19 18:05:28,563 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0ef15a5a788e7e8481c6f8908c6ae5db: 2023-07-19 18:05:28,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:28,565 DEBUG [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1504): Waiting on 2023-07-19 18:05:28,580 DEBUG [Listener at localhost/37991] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44881,1689789924525 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44881,1689789924525 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-19 18:05:28,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44881] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44881,1689789924525 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 18:05:28,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44881] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:59268 deadline: 1689789988583, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44881,1689789924525 aborting 2023-07-19 18:05:28,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44881] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44881,1689789924525 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 18:05:28,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44881] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:59268 deadline: 1689789988693, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,44881,1689789924525 aborting 2023-07-19 18:05:28,765 INFO [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,44881,1689789924525; all regions closed. 2023-07-19 18:05:28,776 DEBUG [RS:0;jenkins-hbase4:44881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:28,777 INFO [RS:0;jenkins-hbase4:44881] regionserver.LeaseManager(133): Closed leases 2023-07-19 18:05:28,777 INFO [RS:0;jenkins-hbase4:44881] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-19 18:05:28,777 INFO [RS:0;jenkins-hbase4:44881] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 18:05:28,777 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 18:05:28,777 INFO [RS:0;jenkins-hbase4:44881] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 18:05:28,777 INFO [RS:0;jenkins-hbase4:44881] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 18:05:28,778 INFO [RS:0;jenkins-hbase4:44881] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44881 2023-07-19 18:05:28,790 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:28,790 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:28,790 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:28,790 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:28,791 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:28,792 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44881,1689789924525] 2023-07-19 18:05:28,792 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44881,1689789924525; numProcessing=1 2023-07-19 18:05:28,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:28,799 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,44881,1689789924525 znode expired, triggering replicatorRemoved event 2023-07-19 18:05:28,892 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 18:05:28,893 INFO [RS:0;jenkins-hbase4:44881] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,44881,1689789924525; zookeeper connection closed. 2023-07-19 18:05:28,893 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:44881-0x1017ec27f410001, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 18:05:28,894 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@37d46fb5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@37d46fb5 2023-07-19 18:05:28,896 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44881,1689789924525 already deleted, retry=false 2023-07-19 18:05:28,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:28,896 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,44881,1689789924525 on jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:28,900 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:44881 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:44881 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 18:05:28,901 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:44881 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:44881 2023-07-19 18:05:28,905 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,44881,1689789924525, splitWal=true, meta=false 2023-07-19 18:05:28,905 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,44881,1689789924525 (carryingMeta=false) jenkins-hbase4.apache.org,44881,1689789924525/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@6bea6a95[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-19 18:05:28,908 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,44881,1689789924525, splitWal=true, meta=false 2023-07-19 18:05:28,909 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,44881,1689789924525 had 1 regions 2023-07-19 18:05:28,911 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,44881,1689789924525, splitWal=true, meta=false, isMeta: false 2023-07-19 18:05:28,913 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting 2023-07-19 18:05:28,915 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,44881,1689789924525 WAL count=1, meta=false 2023-07-19 18:05:28,918 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384}] 2023-07-19 18:05:28,927 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:28,929 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384, worker=jenkins-hbase4.apache.org,42201,1689789924575}] 2023-07-19 18:05:29,096 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42201] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-19 18:05:29,121 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384, size=1.1 K (1148bytes) 2023-07-19 18:05:29,122 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384 2023-07-19 18:05:29,127 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384 after 5ms 2023-07-19 18:05:29,133 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384 took 13ms 2023-07-19 18:05:29,150 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 0ef15a5a788e7e8481c6f8908c6ae5db: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-19 18:05:29,152 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384 so closing down 2023-07-19 18:05:29,152 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 18:05:29,158 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-19 18:05:29,161 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384.temp 2023-07-19 18:05:29,163 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 18:05:29,181 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-19 18:05:29,183 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384.temp to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000013 2023-07-19 18:05:29,183 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 48 ms; skipped=1; WAL=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-19 18:05:29,184 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384, journal: Splitting hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384, size=1.1 K (1148bytes) at 1689789929121Finishing writing output for hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384 so closing down at 1689789929152 (+31 ms)Creating recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384.temp at 1689789929161 (+9 ms)3 split writer threads finished at 1689789929163 (+2 ms)Closed recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689789929181 (+18 ms)Rename recovered edits hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384.temp to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000013 at 1689789929183 (+2 ms)Processed 11 edits across 1 Regions in 48 ms; skipped=1; WAL=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689789929184 (+1 ms) 2023-07-19 18:05:29,184 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384 2023-07-19 18:05:29,187 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-19 18:05:29,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] master.HMaster(3641): Remote procedure done, pid=14 2023-07-19 18:05:29,196 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] wal.WALSplitUtil(141): Moved hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting/jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384 to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs 2023-07-19 18:05:29,202 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-19 18:05:29,202 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384, worker=jenkins-hbase4.apache.org,42201,1689789924575 in 269 msec 2023-07-19 18:05:29,204 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:29,206 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 18:05:29,209 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-19 18:05:29,209 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C44881%2C1689789924525.1689789926384, worker=jenkins-hbase4.apache.org,42201,1689789924575 in 288 msec 2023-07-19 18:05:29,209 DEBUG [hconnection-0x420100ae-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:44881 this server is in the failed servers list 2023-07-19 18:05:29,212 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting dir is empty, no logs to split. 2023-07-19 18:05:29,212 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,44881,1689789924525 WAL count=0, meta=false 2023-07-19 18:05:29,212 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,44881,1689789924525 WAL splitting is done? wals=0, meta=false 2023-07-19 18:05:29,215 INFO [PEWorker-4] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,44881,1689789924525 failed, ignore...File hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,44881,1689789924525-splitting does not exist. 2023-07-19 18:05:29,217 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN}] 2023-07-19 18:05:29,219 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN 2023-07-19 18:05:29,220 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 18:05:29,370 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=0ef15a5a788e7e8481c6f8908c6ae5db, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:29,371 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689789929370"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689789929370"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689789929370"}]},"ts":"1689789929370"} 2023-07-19 18:05:29,374 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 0ef15a5a788e7e8481c6f8908c6ae5db, server=jenkins-hbase4.apache.org,42201,1689789924575}] 2023-07-19 18:05:29,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:29,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0ef15a5a788e7e8481c6f8908c6ae5db, NAME => 't,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:29,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:29,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:29,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:29,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:29,536 INFO [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:29,537 DEBUG [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/f 2023-07-19 18:05:29,537 DEBUG [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/f 2023-07-19 18:05:29,538 INFO [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0ef15a5a788e7e8481c6f8908c6ae5db columnFamilyName f 2023-07-19 18:05:29,539 INFO [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] regionserver.HStore(310): Store=0ef15a5a788e7e8481c6f8908c6ae5db/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:29,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:29,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:29,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000013 2023-07-19 18:05:29,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000013 2023-07-19 18:05:29,556 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0ef15a5a788e7e8481c6f8908c6ae5db 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-19 18:05:29,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/.tmp/f/be519077d9dd4afda70902fa1fabd74d 2023-07-19 18:05:29,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/.tmp/f/be519077d9dd4afda70902fa1fabd74d as hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/f/be519077d9dd4afda70902fa1fabd74d 2023-07-19 18:05:29,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/f/be519077d9dd4afda70902fa1fabd74d, entries=10, sequenceid=13, filesize=4.8 K 2023-07-19 18:05:29,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 0ef15a5a788e7e8481c6f8908c6ae5db in 129ms, sequenceid=13, compaction requested=false; wal=null 2023-07-19 18:05:29,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-19 18:05:29,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/0000000000000000013 2023-07-19 18:05:29,695 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:29,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-19 18:05:29,700 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0ef15a5a788e7e8481c6f8908c6ae5db; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70847869, jitterRate=0.055715516209602356}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 18:05:29,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0ef15a5a788e7e8481c6f8908c6ae5db: 2023-07-19 18:05:29,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db., pid=16, masterSystemTime=1689789929527 2023-07-19 18:05:29,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:29,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:29,705 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=0ef15a5a788e7e8481c6f8908c6ae5db, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:29,705 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689789929705"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689789929705"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689789929705"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689789929705"}]},"ts":"1689789929705"} 2023-07-19 18:05:29,711 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-19 18:05:29,711 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 0ef15a5a788e7e8481c6f8908c6ae5db, server=jenkins-hbase4.apache.org,42201,1689789924575 in 334 msec 2023-07-19 18:05:29,714 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-19 18:05:29,714 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,44881,1689789924525 after splitting done 2023-07-19 18:05:29,714 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN in 494 msec 2023-07-19 18:05:29,714 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,44881,1689789924525 from processing; numProcessing=0 2023-07-19 18:05:29,717 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,44881,1689789924525, splitWal=true, meta=false in 817 msec 2023-07-19 18:05:29,739 DEBUG [Listener at localhost/37991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c11e80b to 127.0.0.1:50575 2023-07-19 18:05:29,740 DEBUG [Listener at localhost/37991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:29,740 INFO [Listener at localhost/37991] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-19 18:05:29,754 INFO [Listener at localhost/37991] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=271 (was 294), OpenFileDescriptor=567 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=175 (was 175), ProcessCount=171 (was 171), AvailableMemoryMB=6462 (was 6499) 2023-07-19 18:05:29,773 INFO [Listener at localhost/37991] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=271, OpenFileDescriptor=567, MaxFileDescriptor=60000, SystemLoadAverage=175, ProcessCount=171, AvailableMemoryMB=6461 2023-07-19 18:05:29,774 INFO [Listener at localhost/37991] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:44881 2023-07-19 18:05:29,789 INFO [Listener at localhost/37991] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 18:05:29,789 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:29,790 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:29,790 INFO [Listener at localhost/37991] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 18:05:29,790 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 18:05:29,790 INFO [Listener at localhost/37991] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 18:05:29,790 INFO [Listener at localhost/37991] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 18:05:29,793 INFO [Listener at localhost/37991] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40393 2023-07-19 18:05:29,794 INFO [Listener at localhost/37991] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 18:05:29,795 DEBUG [Listener at localhost/37991] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 18:05:29,795 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:29,797 INFO [Listener at localhost/37991] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 18:05:29,798 INFO [Listener at localhost/37991] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40393 connecting to ZooKeeper ensemble=127.0.0.1:50575 2023-07-19 18:05:29,801 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:403930x0, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 18:05:29,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:40393-0x1017ec27f410008 connected 2023-07-19 18:05:29,803 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(162): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 18:05:29,803 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(162): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-19 18:05:29,805 DEBUG [Listener at localhost/37991] zookeeper.ZKUtil(164): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 18:05:29,812 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40393 2023-07-19 18:05:29,812 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40393 2023-07-19 18:05:29,812 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40393 2023-07-19 18:05:29,813 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40393 2023-07-19 18:05:29,814 DEBUG [Listener at localhost/37991] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40393 2023-07-19 18:05:29,817 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(951): ClusterId : 675bcfc4-dd2a-40f5-ac0d-070575ea7225 2023-07-19 18:05:29,817 DEBUG [RS:2;jenkins-hbase4:40393] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 18:05:29,820 DEBUG [RS:2;jenkins-hbase4:40393] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 18:05:29,820 DEBUG [RS:2;jenkins-hbase4:40393] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 18:05:29,823 DEBUG [RS:2;jenkins-hbase4:40393] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 18:05:29,824 DEBUG [RS:2;jenkins-hbase4:40393] zookeeper.ReadOnlyZKClient(139): Connect 0x6caba136 to 127.0.0.1:50575 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 18:05:29,833 DEBUG [RS:2;jenkins-hbase4:40393] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@626c52d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 18:05:29,833 DEBUG [RS:2;jenkins-hbase4:40393] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6653ca46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 18:05:29,842 DEBUG [RS:2;jenkins-hbase4:40393] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:40393 2023-07-19 18:05:29,842 INFO [RS:2;jenkins-hbase4:40393] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 18:05:29,842 INFO [RS:2;jenkins-hbase4:40393] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 18:05:29,842 DEBUG [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 18:05:29,843 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42259,1689789923265 with isa=jenkins-hbase4.apache.org/172.31.14.131:40393, startcode=1689789929789 2023-07-19 18:05:29,843 DEBUG [RS:2;jenkins-hbase4:40393] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 18:05:29,847 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32783, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 18:05:29,847 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42259] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:29,848 DEBUG [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1 2023-07-19 18:05:29,848 DEBUG [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46847 2023-07-19 18:05:29,848 DEBUG [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 18:05:29,849 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:29,849 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:29,850 DEBUG [RS:2;jenkins-hbase4:40393] zookeeper.ZKUtil(162): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:29,850 WARN [RS:2;jenkins-hbase4:40393] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 18:05:29,850 INFO [RS:2;jenkins-hbase4:40393] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 18:05:29,850 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40393,1689789929789] 2023-07-19 18:05:29,850 DEBUG [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:29,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:29,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:29,860 DEBUG [RS:2;jenkins-hbase4:40393] zookeeper.ZKUtil(162): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:29,860 DEBUG [RS:2;jenkins-hbase4:40393] zookeeper.ZKUtil(162): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:29,861 DEBUG [RS:2;jenkins-hbase4:40393] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 18:05:29,861 INFO [RS:2;jenkins-hbase4:40393] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 18:05:29,863 INFO [RS:2;jenkins-hbase4:40393] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 18:05:29,864 INFO [RS:2;jenkins-hbase4:40393] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 18:05:29,864 INFO [RS:2;jenkins-hbase4:40393] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:29,866 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 18:05:29,868 INFO [RS:2;jenkins-hbase4:40393] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:29,868 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:29,869 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:29,869 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:29,869 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:29,869 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:29,869 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 18:05:29,869 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:29,869 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:29,869 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:29,869 DEBUG [RS:2;jenkins-hbase4:40393] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 18:05:29,870 INFO [RS:2;jenkins-hbase4:40393] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:29,870 INFO [RS:2;jenkins-hbase4:40393] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:29,870 INFO [RS:2;jenkins-hbase4:40393] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:29,881 INFO [RS:2;jenkins-hbase4:40393] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 18:05:29,882 INFO [RS:2;jenkins-hbase4:40393] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40393,1689789929789-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 18:05:29,893 INFO [RS:2;jenkins-hbase4:40393] regionserver.Replication(203): jenkins-hbase4.apache.org,40393,1689789929789 started 2023-07-19 18:05:29,893 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,40393,1689789929789, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40393, sessionid=0x1017ec27f410008 2023-07-19 18:05:29,893 INFO [Listener at localhost/37991] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:40393,5,FailOnTimeoutGroup] 2023-07-19 18:05:29,893 DEBUG [RS:2;jenkins-hbase4:40393] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 18:05:29,893 DEBUG [RS:2;jenkins-hbase4:40393] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:29,894 DEBUG [RS:2;jenkins-hbase4:40393] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40393,1689789929789' 2023-07-19 18:05:29,894 DEBUG [RS:2;jenkins-hbase4:40393] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 18:05:29,893 INFO [Listener at localhost/37991] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:29,894 INFO [Listener at localhost/37991] client.TestScannerTimeout(177): START ************ test3686b 2023-07-19 18:05:29,894 DEBUG [RS:2;jenkins-hbase4:40393] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 18:05:29,895 DEBUG [RS:2;jenkins-hbase4:40393] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 18:05:29,895 DEBUG [RS:2;jenkins-hbase4:40393] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 18:05:29,895 DEBUG [RS:2;jenkins-hbase4:40393] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:29,895 DEBUG [RS:2;jenkins-hbase4:40393] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40393,1689789929789' 2023-07-19 18:05:29,895 DEBUG [RS:2;jenkins-hbase4:40393] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 18:05:29,895 DEBUG [RS:2;jenkins-hbase4:40393] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 18:05:29,896 DEBUG [RS:2;jenkins-hbase4:40393] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 18:05:29,896 INFO [RS:2;jenkins-hbase4:40393] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 18:05:29,896 INFO [RS:2;jenkins-hbase4:40393] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 18:05:29,898 DEBUG [Listener at localhost/37991] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 18:05:29,898 DEBUG [Listener at localhost/37991] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:29,899 DEBUG [Listener at localhost/37991] zookeeper.ReadOnlyZKClient(139): Connect 0x44c30efb to 127.0.0.1:50575 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 18:05:29,903 DEBUG [Listener at localhost/37991] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37cccf27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 18:05:29,906 DEBUG [hconnection-0x1748fbbb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 18:05:29,910 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43828, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 18:05:29,999 INFO [RS:2;jenkins-hbase4:40393] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40393%2C1689789929789, suffix=, logDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,40393,1689789929789, archiveDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs, maxLogs=32 2023-07-19 18:05:30,028 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK] 2023-07-19 18:05:30,030 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK] 2023-07-19 18:05:30,036 INFO [RS:2;jenkins-hbase4:40393] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,40393,1689789929789/jenkins-hbase4.apache.org%2C40393%2C1689789929789.1689789930001 2023-07-19 18:05:30,039 DEBUG [RS:2;jenkins-hbase4:40393] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK], DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK]] 2023-07-19 18:05:32,220 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-19 18:05:32,277 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-19 18:05:32,278 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-19 18:05:33,445 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 18:05:33,446 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-19 18:05:44,917 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 3903486503052902419 expired clientIPAndPort=172.31.14.131:43828, userName=jenkins, regionInfo=t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:46,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42201] regionserver.RSRpcServices(3094): Client tried to access missing scanner 3903486503052902419 2023-07-19 18:05:46,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42201] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:43828 deadline: 1689790006917, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '3903486503052902419'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-19 18:05:46,919 DEBUG [Listener at localhost/37991] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '3903486503052902419'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '3903486503052902419'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '3903486503052902419'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-19 18:05:46,923 DEBUG [Listener at localhost/37991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44c30efb to 127.0.0.1:50575 2023-07-19 18:05:46,924 DEBUG [Listener at localhost/37991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:46,924 INFO [Listener at localhost/37991] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-19 18:05:46,938 INFO [Listener at localhost/37991] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=294 (was 271) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:2;jenkins-hbase4:40393-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-458194780-172.31.14.131-1689789919967:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50575@0x6caba136 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1191431108.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-563761527_17 at /127.0.0.1:34272 [Receiving block BP-458194780-172.31.14.131-1689789919967:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase4:40393Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1-prefix:jenkins-hbase4.apache.org,40393,1689789929789 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-458194780-172.31.14.131-1689789919967:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50575@0x44c30efb-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50575@0x44c30efb-SendThread(127.0.0.1:50575) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50575@0x6caba136-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50575@0x44c30efb java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1191431108.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:40393 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50575@0x6caba136-SendThread(127.0.0.1:50575) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:46847 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-563761527_17 at /127.0.0.1:54930 [Receiving block BP-458194780-172.31.14.131-1689789919967:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40393 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=571 (was 567) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=152 (was 175), ProcessCount=171 (was 171), AvailableMemoryMB=6387 (was 6461) 2023-07-19 18:05:46,953 INFO [Listener at localhost/37991] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=294, OpenFileDescriptor=571, MaxFileDescriptor=60000, SystemLoadAverage=152, ProcessCount=171, AvailableMemoryMB=6382 2023-07-19 18:05:46,953 INFO [Listener at localhost/37991] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:44881 2023-07-19 18:05:46,954 INFO [Listener at localhost/37991] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,44881,1689789924525 2023-07-19 18:05:46,954 INFO [Listener at localhost/37991] client.TestScannerTimeout(104): START************ test2772 2023-07-19 18:05:46,957 DEBUG [Listener at localhost/37991] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 18:05:46,957 DEBUG [Listener at localhost/37991] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:46,958 DEBUG [Listener at localhost/37991] zookeeper.ReadOnlyZKClient(139): Connect 0x003d8b32 to 127.0.0.1:50575 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 18:05:46,962 DEBUG [Listener at localhost/37991] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d6e31d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 18:05:46,963 ERROR [Listener at localhost/37991] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,42201,1689789924575: die! ***** 2023-07-19 18:05:46,963 ERROR [Listener at localhost/37991] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-19 18:05:46,963 DEBUG [Listener at localhost/37991] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-19 18:05:46,964 DEBUG [Listener at localhost/37991] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-19 18:05:46,969 DEBUG [Listener at localhost/37991] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-19 18:05:46,972 DEBUG [Listener at localhost/37991] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-19 18:05:47,002 INFO [Listener at localhost/37991] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 973078528, "init": 513802240, "max": 2051014656, "used": 269666880 }, "NonHeapMemoryUsage": { "committed": 122642432, "init": 2555904, "max": -1, "used": 120205304 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11956, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 2, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 2, "ProcessCallTime_95th_percentile": 2, "ProcessCallTime_98th_percentile": 2, "ProcessCallTime_99th_percentile": 2, "ProcessCallTime_99.9th_percentile": 2, "ProcessCallTime_TimeRangeCount_0-1": 1, "ProcessCallTime_TimeRangeCount_1-3": 1, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 2, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 2, "TotalCallTime_95th_percentile": 2, "TotalCallTime_98th_percentile": 2, "TotalCallTime_99th_percentile": 2, "TotalCallTime_99.9th_percentile": 2, "TotalCallTime_TimeRangeCount_0-1": 1, "TotalCallTime_TimeRangeCount_1-3": 1, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16661 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:50575", "tag.serverName": "jenkins-hbase4.apache.org,42201,1689789924575", "tag.clusterId": "675bcfc4-dd2a-40f5-ac0d-070575ea7225", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 1, "maxStoreFileCount": 1, "memStoreSize": 3306, "storeFileSize": 4926, "maxStoreFileAge": 16538, "minStoreFileAge": 16538, "avgStoreFileAge": 16538, "numReferenceFiles": 0, "regionServerStartTime": 1689789924575, "averageRegionSize": 2744, "storeFileIndexSize": 320, "staticIndexSize": 31, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 1863680, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 46, "totalRowActionRequestCount": 57, "readRequestCount": 43, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 10, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 22, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 10, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 270, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 21, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 10, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 21, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 270, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.593724915047656, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.152766621213692, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-19 18:05:47,003 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42259] master.MasterRpcServices(612): jenkins-hbase4.apache.org,42201,1689789924575 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,42201,1689789924575: die! ***** 2023-07-19 18:05:47,004 INFO [Listener at localhost/37991] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42201,1689789924575' ***** 2023-07-19 18:05:47,006 INFO [Listener at localhost/37991] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-19 18:05:47,006 INFO [RS:1;jenkins-hbase4:42201] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 18:05:47,007 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 18:05:47,007 INFO [RS:1;jenkins-hbase4:42201] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-19 18:05:47,007 INFO [RS:1;jenkins-hbase4:42201] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-19 18:05:47,007 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(3305): Received CLOSE for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:47,008 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(3305): Received CLOSE for b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:47,008 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:47,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0ef15a5a788e7e8481c6f8908c6ae5db, disabling compactions & flushes 2023-07-19 18:05:47,008 DEBUG [RS:1;jenkins-hbase4:42201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e1736b3 to 127.0.0.1:50575 2023-07-19 18:05:47,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:47,008 DEBUG [RS:1;jenkins-hbase4:42201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:47,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:47,009 INFO [RS:1;jenkins-hbase4:42201] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 18:05:47,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. after waiting 0 ms 2023-07-19 18:05:47,009 INFO [RS:1;jenkins-hbase4:42201] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 18:05:47,009 INFO [RS:1;jenkins-hbase4:42201] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 18:05:47,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:47,009 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-19 18:05:47,014 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-19 18:05:47,016 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 18:05:47,016 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 18:05:47,016 DEBUG [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1478): Online Regions={0ef15a5a788e7e8481c6f8908c6ae5db=t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db., b172f58552c421ef659ce60b0365ef75=hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75., 1588230740=hbase:meta,,1.1588230740} 2023-07-19 18:05:47,016 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 18:05:47,016 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 18:05:47,016 DEBUG [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1504): Waiting on 0ef15a5a788e7e8481c6f8908c6ae5db, 1588230740, b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:47,017 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 18:05:47,018 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 18:05:47,018 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 3228 in region hbase:meta,,1.1588230740 2023-07-19 18:05:47,018 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 18:05:47,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:47,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0ef15a5a788e7e8481c6f8908c6ae5db: 2023-07-19 18:05:47,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:47,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b172f58552c421ef659ce60b0365ef75, disabling compactions & flushes 2023-07-19 18:05:47,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:47,020 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 18:05:47,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:47,020 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 18:05:47,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. after waiting 0 ms 2023-07-19 18:05:47,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:47,020 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-19 18:05:47,020 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 18:05:47,021 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:47,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:47,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b172f58552c421ef659ce60b0365ef75: 2023-07-19 18:05:47,021 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42201] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42201,1689789924575 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 18:05:47,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:47,021 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42201] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:50506 deadline: 1689790007021, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42201,1689789924575 aborting 2023-07-19 18:05:47,041 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 18:05:47,126 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42201] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42201,1689789924575 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 18:05:47,126 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42201] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:50506 deadline: 1689790007125, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,42201,1689789924575 aborting 2023-07-19 18:05:47,207 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-19 18:05:47,207 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-19 18:05:47,217 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,42201,1689789924575; all regions closed. 2023-07-19 18:05:47,230 DEBUG [RS:1;jenkins-hbase4:42201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:47,231 INFO [RS:1;jenkins-hbase4:42201] regionserver.LeaseManager(133): Closed leases 2023-07-19 18:05:47,231 INFO [RS:1;jenkins-hbase4:42201] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-19 18:05:47,231 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 18:05:47,232 INFO [RS:1;jenkins-hbase4:42201] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42201 2023-07-19 18:05:47,237 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:47,237 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 2023-07-19 18:05:47,237 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:47,237 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:47,237 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:47,239 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42201,1689789924575] 2023-07-19 18:05:47,239 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42201,1689789924575; numProcessing=1 2023-07-19 18:05:47,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:47,240 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,42201,1689789924575 znode expired, triggering replicatorRemoved event 2023-07-19 18:05:47,241 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42201,1689789924575 already deleted, retry=false 2023-07-19 18:05:47,241 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,42201,1689789924575 on jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:47,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:47,243 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,42201,1689789924575, splitWal=true, meta=true 2023-07-19 18:05:47,243 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase4.apache.org,42201,1689789924575 (carryingMeta=true) jenkins-hbase4.apache.org,42201,1689789924575/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@64b2249b[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-19 18:05:47,245 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42201,1689789924575, splitWal=true, meta=true 2023-07-19 18:05:47,246 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42201,1689789924575, splitWal=true, meta=true, isMeta: true 2023-07-19 18:05:47,251 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting 2023-07-19 18:05:47,253 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,42201,1689789924575 WAL count=1, meta=true 2023-07-19 18:05:47,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta}] 2023-07-19 18:05:47,257 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:47,258 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta, worker=jenkins-hbase4.apache.org,40393,1689789929789}] 2023-07-19 18:05:47,329 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42201 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42201 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 18:05:47,331 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42201 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42201 2023-07-19 18:05:47,339 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 18:05:47,340 INFO [RS:1;jenkins-hbase4:42201] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42201,1689789924575; zookeeper connection closed. 2023-07-19 18:05:47,340 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:42201-0x1017ec27f410002, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 18:05:47,340 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7bef6fe6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7bef6fe6 2023-07-19 18:05:47,411 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:47,411 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 18:05:47,413 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53434, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 18:05:47,413 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40393] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-19 18:05:47,431 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta, size=4.0 K (4123bytes) 2023-07-19 18:05:47,431 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta 2023-07-19 18:05:47,433 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta after 2ms 2023-07-19 18:05:47,438 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta took 7ms 2023-07-19 18:05:47,443 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-19 18:05:47,445 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta so closing down 2023-07-19 18:05:47,446 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 18:05:47,449 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta.temp 2023-07-19 18:05:47,449 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 18:05:47,460 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-19 18:05:47,463 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta.temp to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-19 18:05:47,463 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 25 ms; skipped=1; WAL=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false 2023-07-19 18:05:47,463 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta, journal: Splitting hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta, size=4.0 K (4123bytes) at 1689789947431Finishing writing output for hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta so closing down at 1689789947446 (+15 ms)Creating recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta.temp at 1689789947449 (+3 ms)3 split writer threads finished at 1689789947449Closed recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689789947461 (+12 ms)Rename recovered edits hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta.temp to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689789947463 (+2 ms)Processed 13 edits across 1 Regions in 25 ms; skipped=1; WAL=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false at 1689789947463 2023-07-19 18:05:47,463 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta 2023-07-19 18:05:47,463 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-19 18:05:47,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] master.HMaster(3641): Remote procedure done, pid=19 2023-07-19 18:05:47,467 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] wal.WALSplitUtil(141): Moved hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs 2023-07-19 18:05:47,471 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-19 18:05:47,471 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta, worker=jenkins-hbase4.apache.org,40393,1689789929789 in 210 msec 2023-07-19 18:05:47,473 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:47,475 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-19 18:05:47,475 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C42201%2C1689789924575.meta.1689789926598.meta, worker=jenkins-hbase4.apache.org,40393,1689789929789 in 219 msec 2023-07-19 18:05:47,478 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting dir is empty, no logs to split. 2023-07-19 18:05:47,479 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,42201,1689789924575 WAL count=0, meta=true 2023-07-19 18:05:47,479 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42201,1689789924575 WAL splitting is done? wals=0, meta=true 2023-07-19 18:05:47,480 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-19 18:05:47,481 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-19 18:05:47,483 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 18:05:47,633 DEBUG [jenkins-hbase4:42259] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 18:05:47,633 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40393,1689789929789, state=OPENING 2023-07-19 18:05:47,635 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 18:05:47,635 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 18:05:47,635 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40393,1689789929789}] 2023-07-19 18:05:47,636 WARN [ReadOnlyZKClient-127.0.0.1:50575@0x003d8b32] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-19 18:05:47,637 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 18:05:47,640 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 18:05:47,641 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40393] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:53436 deadline: 1689790007641, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:47,798 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-19 18:05:47,798 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 18:05:47,800 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40393%2C1689789929789.meta, suffix=.meta, logDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,40393,1689789929789, archiveDir=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs, maxLogs=32 2023-07-19 18:05:47,824 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK] 2023-07-19 18:05:47,825 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK] 2023-07-19 18:05:47,836 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,40393,1689789929789/jenkins-hbase4.apache.org%2C40393%2C1689789929789.meta.1689789947801.meta 2023-07-19 18:05:47,836 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45237,DS-a7866e61-9d1d-4f0c-b3b0-946be2e3a0fe,DISK], DatanodeInfoWithStorage[127.0.0.1:38129,DS-448ed352-4365-4f01-ad82-b1ffa623ca8c,DISK]] 2023-07-19 18:05:47,836 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:47,837 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-19 18:05:47,837 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-19 18:05:47,838 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-19 18:05:47,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-19 18:05:47,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:47,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-19 18:05:47,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-19 18:05:47,845 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 18:05:47,848 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/info 2023-07-19 18:05:47,848 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/info 2023-07-19 18:05:47,848 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 18:05:47,849 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:47,850 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 18:05:47,851 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/rep_barrier 2023-07-19 18:05:47,851 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/rep_barrier 2023-07-19 18:05:47,852 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 18:05:47,853 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:47,853 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 18:05:47,855 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/table 2023-07-19 18:05:47,855 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/table 2023-07-19 18:05:47,857 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 18:05:47,859 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:47,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740 2023-07-19 18:05:47,866 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740 2023-07-19 18:05:47,868 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-19 18:05:47,879 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-19 18:05:47,880 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.15 KB heapSize=6.27 KB 2023-07-19 18:05:47,962 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/.tmp/info/3f7ac4822bcb41dc9b169dbd8ca0847d 2023-07-19 18:05:48,005 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/.tmp/table/8d323ab4fd124d28bd57b32e3c0437e2 2023-07-19 18:05:48,017 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/.tmp/info/3f7ac4822bcb41dc9b169dbd8ca0847d as hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/info/3f7ac4822bcb41dc9b169dbd8ca0847d 2023-07-19 18:05:48,030 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/info/3f7ac4822bcb41dc9b169dbd8ca0847d, entries=28, sequenceid=15, filesize=7.7 K 2023-07-19 18:05:48,033 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/.tmp/table/8d323ab4fd124d28bd57b32e3c0437e2 as hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/table/8d323ab4fd124d28bd57b32e3c0437e2 2023-07-19 18:05:48,044 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/table/8d323ab4fd124d28bd57b32e3c0437e2, entries=4, sequenceid=15, filesize=4.7 K 2023-07-19 18:05:48,044 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.15 KB/3228, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 164ms, sequenceid=15, compaction requested=false; wal=null 2023-07-19 18:05:48,046 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-19 18:05:48,050 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 18:05:48,053 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 18:05:48,059 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-19 18:05:48,064 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61157617, jitterRate=-0.08868049085140228}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 18:05:48,064 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 18:05:48,073 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689789947791 2023-07-19 18:05:48,077 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-19 18:05:48,078 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-19 18:05:48,080 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40393,1689789929789, state=OPEN 2023-07-19 18:05:48,083 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 18:05:48,083 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 18:05:48,096 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-19 18:05:48,096 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40393,1689789929789 in 448 msec 2023-07-19 18:05:48,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-19 18:05:48,101 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,42201,1689789924575 had 3 regions 2023-07-19 18:05:48,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 616 msec 2023-07-19 18:05:48,102 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,42201,1689789924575, splitWal=true, meta=true, isMeta: false 2023-07-19 18:05:48,105 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,42201,1689789924575 WAL count=1, meta=false 2023-07-19 18:05:48,105 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384}] 2023-07-19 18:05:48,109 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:48,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384, worker=jenkins-hbase4.apache.org,40393,1689789929789}] 2023-07-19 18:05:48,156 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42201 this server is in the failed servers list 2023-07-19 18:05:48,264 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40393] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-19 18:05:48,271 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42201 this server is in the failed servers list 2023-07-19 18:05:48,293 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384, size=899 (899bytes) 2023-07-19 18:05:48,293 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384 2023-07-19 18:05:48,294 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384 after 1ms 2023-07-19 18:05:48,297 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384 took 4ms 2023-07-19 18:05:48,302 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for b172f58552c421ef659ce60b0365ef75: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-19 18:05:48,305 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 0ef15a5a788e7e8481c6f8908c6ae5db: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-19 18:05:48,305 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384 so closing down 2023-07-19 18:05:48,305 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 18:05:48,309 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384.temp 2023-07-19 18:05:48,309 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 18:05:48,331 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-19 18:05:48,333 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384.temp to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000005 2023-07-19 18:05:48,333 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 35 ms; skipped=2; WAL=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384, size=899, length=899, corrupted=false, cancelled=false 2023-07-19 18:05:48,333 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384, journal: Splitting hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384, size=899 (899bytes) at 1689789948293Finishing writing output for hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384 so closing down at 1689789948305 (+12 ms)Creating recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384.temp at 1689789948309 (+4 ms)3 split writer threads finished at 1689789948309Closed recovered edits writer path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689789948331 (+22 ms)Rename recovered edits hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384.temp to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000005 at 1689789948333 (+2 ms)Processed 4 edits across 1 Regions in 35 ms; skipped=2; WAL=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384, size=899, length=899, corrupted=false, cancelled=false at 1689789948333 2023-07-19 18:05:48,333 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384 2023-07-19 18:05:48,333 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-19 18:05:48,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] master.HMaster(3641): Remote procedure done, pid=23 2023-07-19 18:05:48,338 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42259] wal.WALSplitUtil(141): Moved hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting/jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384 to hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs 2023-07-19 18:05:48,342 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-19 18:05:48,343 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384, worker=jenkins-hbase4.apache.org,40393,1689789929789 in 230 msec 2023-07-19 18:05:48,346 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:48,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-19 18:05:48,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C42201%2C1689789924575.1689789926384, worker=jenkins-hbase4.apache.org,40393,1689789929789 in 241 msec 2023-07-19 18:05:48,352 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/WALs/jenkins-hbase4.apache.org,42201,1689789924575-splitting dir is empty, no logs to split. 2023-07-19 18:05:48,352 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,42201,1689789924575 WAL count=0, meta=false 2023-07-19 18:05:48,352 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,42201,1689789924575 WAL splitting is done? wals=0, meta=false 2023-07-19 18:05:48,355 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b172f58552c421ef659ce60b0365ef75, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN}] 2023-07-19 18:05:48,357 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b172f58552c421ef659ce60b0365ef75, ASSIGN 2023-07-19 18:05:48,357 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN 2023-07-19 18:05:48,358 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=b172f58552c421ef659ce60b0365ef75, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 18:05:48,358 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 18:05:48,479 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42201 this server is in the failed servers list 2023-07-19 18:05:48,508 DEBUG [jenkins-hbase4:42259] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 18:05:48,508 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=0ef15a5a788e7e8481c6f8908c6ae5db, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:48,509 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=b172f58552c421ef659ce60b0365ef75, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:48,509 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689789948508"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689789948508"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689789948508"}]},"ts":"1689789948508"} 2023-07-19 18:05:48,509 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689789948508"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689789948508"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689789948508"}]},"ts":"1689789948508"} 2023-07-19 18:05:48,510 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42201 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42201 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 18:05:48,516 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42201 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42201 2023-07-19 18:05:48,615 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 18:05:48,617 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53450, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 18:05:48,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; OpenRegionProcedure 0ef15a5a788e7e8481c6f8908c6ae5db, server=jenkins-hbase4.apache.org,40393,1689789929789}] 2023-07-19 18:05:48,624 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure b172f58552c421ef659ce60b0365ef75, server=jenkins-hbase4.apache.org,40393,1689789929789}] 2023-07-19 18:05:48,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:48,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0ef15a5a788e7e8481c6f8908c6ae5db, NAME => 't,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:48,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:48,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:48,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:48,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:48,789 INFO [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:48,790 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:42201 this server is in the failed servers list 2023-07-19 18:05:48,791 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:48,791 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b172f58552c421ef659ce60b0365ef75, NAME => 'hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.', STARTKEY => '', ENDKEY => ''} 2023-07-19 18:05:48,791 DEBUG [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/f 2023-07-19 18:05:48,791 DEBUG [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/f 2023-07-19 18:05:48,791 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:48,791 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 18:05:48,791 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:48,791 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:48,792 INFO [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0ef15a5a788e7e8481c6f8908c6ae5db columnFamilyName f 2023-07-19 18:05:48,800 INFO [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:48,805 DEBUG [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/info 2023-07-19 18:05:48,805 DEBUG [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/info 2023-07-19 18:05:48,806 INFO [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b172f58552c421ef659ce60b0365ef75 columnFamilyName info 2023-07-19 18:05:48,807 INFO [StoreOpener-b172f58552c421ef659ce60b0365ef75-1] regionserver.HStore(310): Store=b172f58552c421ef659ce60b0365ef75/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:48,808 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:48,813 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:48,814 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000005 2023-07-19 18:05:48,815 DEBUG [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] regionserver.HStore(539): loaded hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/f/be519077d9dd4afda70902fa1fabd74d 2023-07-19 18:05:48,816 INFO [StoreOpener-0ef15a5a788e7e8481c6f8908c6ae5db-1] regionserver.HStore(310): Store=0ef15a5a788e7e8481c6f8908c6ae5db/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 18:05:48,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:48,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:48,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:48,826 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000005 2023-07-19 18:05:48,826 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b172f58552c421ef659ce60b0365ef75 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-19 18:05:48,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-19 18:05:48,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0ef15a5a788e7e8481c6f8908c6ae5db; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62367791, jitterRate=-0.07064749300479889}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 18:05:48,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0ef15a5a788e7e8481c6f8908c6ae5db: 2023-07-19 18:05:48,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db., pid=26, masterSystemTime=1689789948780 2023-07-19 18:05:48,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:48,846 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:48,846 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=0ef15a5a788e7e8481c6f8908c6ae5db, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:48,847 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689789948846"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689789948846"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689789948846"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689789948846"}]},"ts":"1689789948846"} 2023-07-19 18:05:48,861 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-19 18:05:48,861 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; OpenRegionProcedure 0ef15a5a788e7e8481c6f8908c6ae5db, server=jenkins-hbase4.apache.org,40393,1689789929789 in 230 msec 2023-07-19 18:05:48,867 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=0ef15a5a788e7e8481c6f8908c6ae5db, ASSIGN in 506 msec 2023-07-19 18:05:48,871 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/.tmp/info/0a35286112b94ddfb0884a5481cba845 2023-07-19 18:05:48,882 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/.tmp/info/0a35286112b94ddfb0884a5481cba845 as hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/info/0a35286112b94ddfb0884a5481cba845 2023-07-19 18:05:48,894 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/info/0a35286112b94ddfb0884a5481cba845, entries=2, sequenceid=5, filesize=4.8 K 2023-07-19 18:05:48,895 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for b172f58552c421ef659ce60b0365ef75 in 69ms, sequenceid=5, compaction requested=false; wal=null 2023-07-19 18:05:48,896 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/0000000000000000005 2023-07-19 18:05:48,901 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:48,906 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-19 18:05:48,907 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b172f58552c421ef659ce60b0365ef75; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74575429, jitterRate=0.11126048862934113}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 18:05:48,908 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b172f58552c421ef659ce60b0365ef75: 2023-07-19 18:05:48,909 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75., pid=27, masterSystemTime=1689789948780 2023-07-19 18:05:48,911 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:48,911 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:48,912 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=b172f58552c421ef659ce60b0365ef75, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:48,912 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689789948912"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689789948912"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689789948912"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689789948912"}]},"ts":"1689789948912"} 2023-07-19 18:05:48,918 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-07-19 18:05:48,918 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure b172f58552c421ef659ce60b0365ef75, server=jenkins-hbase4.apache.org,40393,1689789929789 in 291 msec 2023-07-19 18:05:48,921 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=17 2023-07-19 18:05:48,922 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,42201,1689789924575 after splitting done 2023-07-19 18:05:48,922 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=b172f58552c421ef659ce60b0365ef75, ASSIGN in 563 msec 2023-07-19 18:05:48,922 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,42201,1689789924575 from processing; numProcessing=0 2023-07-19 18:05:48,925 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,42201,1689789924575, splitWal=true, meta=true in 1.6810 sec 2023-07-19 18:05:49,306 DEBUG [Listener at localhost/37991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x003d8b32 to 127.0.0.1:50575 2023-07-19 18:05:49,306 DEBUG [Listener at localhost/37991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:49,306 INFO [Listener at localhost/37991] client.TestScannerTimeout(124): END ************ test2772 2023-07-19 18:05:49,325 INFO [Listener at localhost/37991] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=277 (was 294), OpenFileDescriptor=583 (was 571) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=164 (was 152) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=6285 (was 6382) 2023-07-19 18:05:49,326 INFO [Listener at localhost/37991] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-19 18:05:49,327 INFO [Listener at localhost/37991] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-19 18:05:49,327 DEBUG [Listener at localhost/37991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4789b34a to 127.0.0.1:50575 2023-07-19 18:05:49,327 DEBUG [Listener at localhost/37991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:49,327 DEBUG [Listener at localhost/37991] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-19 18:05:49,327 DEBUG [Listener at localhost/37991] util.JVMClusterUtil(257): Found active master hash=1565997889, stopped=false 2023-07-19 18:05:49,328 INFO [Listener at localhost/37991] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:49,335 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-19 18:05:49,335 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-19 18:05:49,336 INFO [Listener at localhost/37991] procedure2.ProcedureExecutor(629): Stopping 2023-07-19 18:05:49,336 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:49,336 DEBUG [Listener at localhost/37991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x696f7b15 to 127.0.0.1:50575 2023-07-19 18:05:49,336 DEBUG [Listener at localhost/37991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:49,337 INFO [Listener at localhost/37991] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,40393,1689789929789' ***** 2023-07-19 18:05:49,337 INFO [Listener at localhost/37991] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-19 18:05:49,337 INFO [RS:2;jenkins-hbase4:40393] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 18:05:49,337 INFO [RS:2;jenkins-hbase4:40393] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-19 18:05:49,337 INFO [RS:2;jenkins-hbase4:40393] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-19 18:05:49,337 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(3305): Received CLOSE for 0ef15a5a788e7e8481c6f8908c6ae5db 2023-07-19 18:05:49,338 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 18:05:49,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 18:05:49,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 18:05:49,338 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(3305): Received CLOSE for b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:49,339 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:49,339 DEBUG [RS:2;jenkins-hbase4:40393] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6caba136 to 127.0.0.1:50575 2023-07-19 18:05:49,339 DEBUG [RS:2;jenkins-hbase4:40393] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:49,339 INFO [RS:2;jenkins-hbase4:40393] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 18:05:49,339 INFO [RS:2;jenkins-hbase4:40393] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 18:05:49,339 INFO [RS:2;jenkins-hbase4:40393] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 18:05:49,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0ef15a5a788e7e8481c6f8908c6ae5db, disabling compactions & flushes 2023-07-19 18:05:49,340 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-19 18:05:49,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:49,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:49,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. after waiting 0 ms 2023-07-19 18:05:49,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:49,340 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-19 18:05:49,340 DEBUG [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1478): Online Regions={0ef15a5a788e7e8481c6f8908c6ae5db=t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db., b172f58552c421ef659ce60b0365ef75=hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75., 1588230740=hbase:meta,,1.1588230740} 2023-07-19 18:05:49,340 DEBUG [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1504): Waiting on 0ef15a5a788e7e8481c6f8908c6ae5db, 1588230740, b172f58552c421ef659ce60b0365ef75 2023-07-19 18:05:49,341 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 18:05:49,341 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 18:05:49,341 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 18:05:49,341 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 18:05:49,341 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 18:05:49,341 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-19 18:05:49,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/default/t/0ef15a5a788e7e8481c6f8908c6ae5db/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-19 18:05:49,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:49,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0ef15a5a788e7e8481c6f8908c6ae5db: 2023-07-19 18:05:49,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689789927807.0ef15a5a788e7e8481c6f8908c6ae5db. 2023-07-19 18:05:49,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b172f58552c421ef659ce60b0365ef75, disabling compactions & flushes 2023-07-19 18:05:49,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:49,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:49,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. after waiting 0 ms 2023-07-19 18:05:49,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:49,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/namespace/b172f58552c421ef659ce60b0365ef75/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-19 18:05:49,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:49,376 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b172f58552c421ef659ce60b0365ef75: 2023-07-19 18:05:49,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689789926947.b172f58552c421ef659ce60b0365ef75. 2023-07-19 18:05:49,401 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/.tmp/info/c9a84f05e44a44a5afa236e43b935492 2023-07-19 18:05:49,411 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/.tmp/info/c9a84f05e44a44a5afa236e43b935492 as hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/info/c9a84f05e44a44a5afa236e43b935492 2023-07-19 18:05:49,421 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/info/c9a84f05e44a44a5afa236e43b935492, entries=16, sequenceid=22, filesize=6.4 K 2023-07-19 18:05:49,422 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 81ms, sequenceid=22, compaction requested=false 2023-07-19 18:05:49,436 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-19 18:05:49,437 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 18:05:49,438 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 18:05:49,438 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 18:05:49,438 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-19 18:05:49,540 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,40393,1689789929789; all regions closed. 2023-07-19 18:05:49,548 DEBUG [RS:2;jenkins-hbase4:40393] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs 2023-07-19 18:05:49,548 INFO [RS:2;jenkins-hbase4:40393] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40393%2C1689789929789.meta:.meta(num 1689789947801) 2023-07-19 18:05:49,554 DEBUG [RS:2;jenkins-hbase4:40393] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/oldWALs 2023-07-19 18:05:49,554 INFO [RS:2;jenkins-hbase4:40393] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40393%2C1689789929789:(num 1689789930001) 2023-07-19 18:05:49,554 DEBUG [RS:2;jenkins-hbase4:40393] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:49,554 INFO [RS:2;jenkins-hbase4:40393] regionserver.LeaseManager(133): Closed leases 2023-07-19 18:05:49,555 INFO [RS:2;jenkins-hbase4:40393] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-19 18:05:49,555 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 18:05:49,556 INFO [RS:2;jenkins-hbase4:40393] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40393 2023-07-19 18:05:49,558 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:49,559 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40393,1689789929789] 2023-07-19 18:05:49,559 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40393,1689789929789 2023-07-19 18:05:49,559 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40393,1689789929789; numProcessing=1 2023-07-19 18:05:49,559 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 18:05:49,562 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40393,1689789929789 already deleted, retry=false 2023-07-19 18:05:49,562 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40393,1689789929789 expired; onlineServers=0 2023-07-19 18:05:49,562 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42259,1689789923265' ***** 2023-07-19 18:05:49,562 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-19 18:05:49,563 DEBUG [M:0;jenkins-hbase4:42259] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c6a317a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 18:05:49,563 INFO [M:0;jenkins-hbase4:42259] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,42259,1689789923265 2023-07-19 18:05:49,563 INFO [M:0;jenkins-hbase4:42259] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,42259,1689789923265; all regions closed. 2023-07-19 18:05:49,563 DEBUG [M:0;jenkins-hbase4:42259] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 18:05:49,563 DEBUG [M:0;jenkins-hbase4:42259] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-19 18:05:49,563 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-19 18:05:49,563 DEBUG [M:0;jenkins-hbase4:42259] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-19 18:05:49,563 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689789925960] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689789925960,5,FailOnTimeoutGroup] 2023-07-19 18:05:49,563 INFO [M:0;jenkins-hbase4:42259] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-19 18:05:49,563 INFO [M:0;jenkins-hbase4:42259] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-19 18:05:49,564 INFO [M:0;jenkins-hbase4:42259] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-19 18:05:49,564 DEBUG [M:0;jenkins-hbase4:42259] master.HMaster(1512): Stopping service threads 2023-07-19 18:05:49,564 INFO [M:0;jenkins-hbase4:42259] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-19 18:05:49,564 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689789925955] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689789925955,5,FailOnTimeoutGroup] 2023-07-19 18:05:49,565 ERROR [M:0;jenkins-hbase4:42259] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (973461218) connection to localhost/127.0.0.1:46847 from jenkins,5,PEWorkerGroup] 2023-07-19 18:05:49,565 INFO [M:0;jenkins-hbase4:42259] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-19 18:05:49,565 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-19 18:05:49,565 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-19 18:05:49,565 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 18:05:49,566 DEBUG [M:0;jenkins-hbase4:42259] zookeeper.ZKUtil(398): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-19 18:05:49,566 WARN [M:0;jenkins-hbase4:42259] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-19 18:05:49,566 INFO [M:0;jenkins-hbase4:42259] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-19 18:05:49,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 18:05:49,566 INFO [M:0;jenkins-hbase4:42259] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-19 18:05:49,567 DEBUG [M:0;jenkins-hbase4:42259] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-19 18:05:49,567 INFO [M:0;jenkins-hbase4:42259] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 18:05:49,567 DEBUG [M:0;jenkins-hbase4:42259] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 18:05:49,567 DEBUG [M:0;jenkins-hbase4:42259] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-19 18:05:49,567 DEBUG [M:0;jenkins-hbase4:42259] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 18:05:49,567 INFO [M:0;jenkins-hbase4:42259] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=90.79 KB heapSize=109.53 KB 2023-07-19 18:05:49,590 INFO [M:0;jenkins-hbase4:42259] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=90.79 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ae5dcefab77641b0a97a106ca60cfaef 2023-07-19 18:05:49,602 DEBUG [M:0;jenkins-hbase4:42259] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ae5dcefab77641b0a97a106ca60cfaef as hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ae5dcefab77641b0a97a106ca60cfaef 2023-07-19 18:05:49,609 INFO [M:0;jenkins-hbase4:42259] regionserver.HStore(1080): Added hdfs://localhost:46847/user/jenkins/test-data/fe545155-3d98-babc-56c1-7ce3035035a1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ae5dcefab77641b0a97a106ca60cfaef, entries=27, sequenceid=212, filesize=8.2 K 2023-07-19 18:05:49,610 INFO [M:0;jenkins-hbase4:42259] regionserver.HRegion(2948): Finished flush of dataSize ~90.79 KB/92965, heapSize ~109.52 KB/112144, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 43ms, sequenceid=212, compaction requested=false 2023-07-19 18:05:49,611 INFO [M:0;jenkins-hbase4:42259] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 18:05:49,611 DEBUG [M:0;jenkins-hbase4:42259] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 18:05:49,620 INFO [M:0;jenkins-hbase4:42259] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-19 18:05:49,620 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 18:05:49,621 INFO [M:0;jenkins-hbase4:42259] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42259 2023-07-19 18:05:49,623 DEBUG [M:0;jenkins-hbase4:42259] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,42259,1689789923265 already deleted, retry=false 2023-07-19 18:05:49,812 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 18:05:49,812 INFO [M:0;jenkins-hbase4:42259] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42259,1689789923265; zookeeper connection closed. 2023-07-19 18:05:49,812 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): master:42259-0x1017ec27f410000, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 18:05:49,872 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 18:05:49,912 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 18:05:49,912 INFO [RS:2;jenkins-hbase4:40393] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,40393,1689789929789; zookeeper connection closed. 2023-07-19 18:05:49,912 DEBUG [Listener at localhost/37991-EventThread] zookeeper.ZKWatcher(600): regionserver:40393-0x1017ec27f410008, quorum=127.0.0.1:50575, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 18:05:49,913 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6692fa68] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6692fa68 2023-07-19 18:05:49,913 INFO [Listener at localhost/37991] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-19 18:05:49,913 WARN [Listener at localhost/37991] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-19 18:05:49,918 INFO [Listener at localhost/37991] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 18:05:50,025 WARN [BP-458194780-172.31.14.131-1689789919967 heartbeating to localhost/127.0.0.1:46847] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-19 18:05:50,026 WARN [BP-458194780-172.31.14.131-1689789919967 heartbeating to localhost/127.0.0.1:46847] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-458194780-172.31.14.131-1689789919967 (Datanode Uuid 2190df25-9c9e-4352-9e82-1a6fced49c9e) service to localhost/127.0.0.1:46847 2023-07-19 18:05:50,028 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe/dfs/data/data3/current/BP-458194780-172.31.14.131-1689789919967] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 18:05:50,029 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe/dfs/data/data4/current/BP-458194780-172.31.14.131-1689789919967] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 18:05:50,030 WARN [Listener at localhost/37991] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-19 18:05:50,041 INFO [Listener at localhost/37991] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 18:05:50,147 WARN [BP-458194780-172.31.14.131-1689789919967 heartbeating to localhost/127.0.0.1:46847] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-19 18:05:50,147 WARN [BP-458194780-172.31.14.131-1689789919967 heartbeating to localhost/127.0.0.1:46847] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-458194780-172.31.14.131-1689789919967 (Datanode Uuid 444984d7-9db9-44ec-a6e3-b3c00117edd9) service to localhost/127.0.0.1:46847 2023-07-19 18:05:50,147 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe/dfs/data/data1/current/BP-458194780-172.31.14.131-1689789919967] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 18:05:50,148 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6c4a448c-396c-33aa-9c23-a9c483544f80/cluster_aa26991e-6122-0855-02d6-57860c5ec1fe/dfs/data/data2/current/BP-458194780-172.31.14.131-1689789919967] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 18:05:50,198 INFO [Listener at localhost/37991] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 18:05:50,313 INFO [Listener at localhost/37991] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-19 18:05:50,358 INFO [Listener at localhost/37991] hbase.HBaseTestingUtility(1293): Minicluster is down