2023-07-15 02:05:35,510 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-15 02:05:35,737 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d 2023-07-15 02:05:35,751 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-15 02:05:35,751 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0, deleteOnExit=true 2023-07-15 02:05:35,752 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-15 02:05:35,752 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/test.cache.data in system properties and HBase conf 2023-07-15 02:05:35,753 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/hadoop.tmp.dir in system properties and HBase conf 2023-07-15 02:05:35,753 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/hadoop.log.dir in system properties and HBase conf 2023-07-15 02:05:35,754 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-15 02:05:35,755 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-15 02:05:35,755 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-15 02:05:35,868 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-15 02:05:36,312 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-15 02:05:36,317 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-15 02:05:36,318 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-15 02:05:36,318 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-15 02:05:36,319 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 02:05:36,319 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-15 02:05:36,320 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-15 02:05:36,320 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 02:05:36,320 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 02:05:36,321 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-15 02:05:36,321 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/nfs.dump.dir in system properties and HBase conf 2023-07-15 02:05:36,322 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/java.io.tmpdir in system properties and HBase conf 2023-07-15 02:05:36,322 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 02:05:36,322 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-15 02:05:36,323 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-15 02:05:36,861 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 02:05:36,865 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 02:05:37,138 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-15 02:05:37,295 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-15 02:05:37,316 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 02:05:37,362 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 02:05:37,393 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/java.io.tmpdir/Jetty_localhost_33033_hdfs____rez8wq/webapp 2023-07-15 02:05:37,566 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33033 2023-07-15 02:05:37,576 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 02:05:37,576 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 02:05:37,978 WARN [Listener at localhost/45773] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 02:05:38,094 WARN [Listener at localhost/45773] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 02:05:38,115 WARN [Listener at localhost/45773] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 02:05:38,120 INFO [Listener at localhost/45773] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 02:05:38,125 INFO [Listener at localhost/45773] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/java.io.tmpdir/Jetty_localhost_44995_datanode____x7pblo/webapp 2023-07-15 02:05:38,223 INFO [Listener at localhost/45773] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44995 2023-07-15 02:05:38,521 WARN [Listener at localhost/46237] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 02:05:38,532 WARN [Listener at localhost/46237] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 02:05:38,535 WARN [Listener at localhost/46237] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 02:05:38,538 INFO [Listener at localhost/46237] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 02:05:38,543 INFO [Listener at localhost/46237] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/java.io.tmpdir/Jetty_localhost_37303_datanode____.83sj9j/webapp 2023-07-15 02:05:38,655 INFO [Listener at localhost/46237] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37303 2023-07-15 02:05:38,664 WARN [Listener at localhost/35175] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 02:05:38,982 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4e0438302850cb68: Processing first storage report for DS-599b7d38-4356-430b-807e-f9689ed79346 from datanode ef4d3625-7d61-4442-b5d5-bb90b5c13299 2023-07-15 02:05:38,983 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4e0438302850cb68: from storage DS-599b7d38-4356-430b-807e-f9689ed79346 node DatanodeRegistration(127.0.0.1:34079, datanodeUuid=ef4d3625-7d61-4442-b5d5-bb90b5c13299, infoPort=36555, infoSecurePort=0, ipcPort=35175, storageInfo=lv=-57;cid=testClusterID;nsid=1415870821;c=1689386736936), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-15 02:05:38,984 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xffc1e6e3008d2714: Processing first storage report for DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5 from datanode 5c995cd3-d4fc-4681-ac43-212a4a8e9da5 2023-07-15 02:05:38,984 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xffc1e6e3008d2714: from storage DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5 node DatanodeRegistration(127.0.0.1:43351, datanodeUuid=5c995cd3-d4fc-4681-ac43-212a4a8e9da5, infoPort=44945, infoSecurePort=0, ipcPort=46237, storageInfo=lv=-57;cid=testClusterID;nsid=1415870821;c=1689386736936), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 02:05:38,984 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4e0438302850cb68: Processing first storage report for DS-adcd27d3-b87f-4306-878a-b623d417eb37 from datanode ef4d3625-7d61-4442-b5d5-bb90b5c13299 2023-07-15 02:05:38,984 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4e0438302850cb68: from storage DS-adcd27d3-b87f-4306-878a-b623d417eb37 node DatanodeRegistration(127.0.0.1:34079, datanodeUuid=ef4d3625-7d61-4442-b5d5-bb90b5c13299, infoPort=36555, infoSecurePort=0, ipcPort=35175, storageInfo=lv=-57;cid=testClusterID;nsid=1415870821;c=1689386736936), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 02:05:38,984 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xffc1e6e3008d2714: Processing first storage report for DS-40352353-d3b0-4408-9ecb-8be1044d449c from datanode 5c995cd3-d4fc-4681-ac43-212a4a8e9da5 2023-07-15 02:05:38,984 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xffc1e6e3008d2714: from storage DS-40352353-d3b0-4408-9ecb-8be1044d449c node DatanodeRegistration(127.0.0.1:43351, datanodeUuid=5c995cd3-d4fc-4681-ac43-212a4a8e9da5, infoPort=44945, infoSecurePort=0, ipcPort=46237, storageInfo=lv=-57;cid=testClusterID;nsid=1415870821;c=1689386736936), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 02:05:39,061 DEBUG [Listener at localhost/35175] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d 2023-07-15 02:05:39,122 INFO [Listener at localhost/35175] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/zookeeper_0, clientPort=55392, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-15 02:05:39,137 INFO [Listener at localhost/35175] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=55392 2023-07-15 02:05:39,148 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:39,151 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:39,803 INFO [Listener at localhost/35175] util.FSUtils(471): Created version file at hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd with version=8 2023-07-15 02:05:39,803 INFO [Listener at localhost/35175] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/hbase-staging 2023-07-15 02:05:40,130 INFO [Listener at localhost/35175] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-15 02:05:40,583 INFO [Listener at localhost/35175] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 02:05:40,614 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:40,615 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:40,615 INFO [Listener at localhost/35175] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 02:05:40,615 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:40,615 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 02:05:40,753 INFO [Listener at localhost/35175] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 02:05:40,824 DEBUG [Listener at localhost/35175] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-15 02:05:40,920 INFO [Listener at localhost/35175] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45169 2023-07-15 02:05:40,930 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:40,932 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:40,953 INFO [Listener at localhost/35175] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45169 connecting to ZooKeeper ensemble=127.0.0.1:55392 2023-07-15 02:05:40,992 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:451690x0, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 02:05:40,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:45169-0x10166ba68a70000 connected 2023-07-15 02:05:41,020 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 02:05:41,021 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 02:05:41,024 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 02:05:41,032 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45169 2023-07-15 02:05:41,032 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45169 2023-07-15 02:05:41,033 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45169 2023-07-15 02:05:41,033 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45169 2023-07-15 02:05:41,033 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45169 2023-07-15 02:05:41,039 INFO [Listener at localhost/35175] master.HMaster(444): hbase.rootdir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd, hbase.cluster.distributed=false 2023-07-15 02:05:41,105 INFO [Listener at localhost/35175] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 02:05:41,105 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:41,105 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:41,105 INFO [Listener at localhost/35175] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 02:05:41,105 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:41,105 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 02:05:41,110 INFO [Listener at localhost/35175] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 02:05:41,113 INFO [Listener at localhost/35175] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39885 2023-07-15 02:05:41,115 INFO [Listener at localhost/35175] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 02:05:41,121 DEBUG [Listener at localhost/35175] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 02:05:41,123 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:41,125 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:41,126 INFO [Listener at localhost/35175] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39885 connecting to ZooKeeper ensemble=127.0.0.1:55392 2023-07-15 02:05:41,129 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:398850x0, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 02:05:41,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39885-0x10166ba68a70001 connected 2023-07-15 02:05:41,131 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 02:05:41,132 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 02:05:41,133 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 02:05:41,133 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39885 2023-07-15 02:05:41,134 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39885 2023-07-15 02:05:41,134 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39885 2023-07-15 02:05:41,134 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39885 2023-07-15 02:05:41,135 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39885 2023-07-15 02:05:41,148 INFO [Listener at localhost/35175] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 02:05:41,148 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:41,149 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:41,149 INFO [Listener at localhost/35175] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 02:05:41,149 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:41,149 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 02:05:41,150 INFO [Listener at localhost/35175] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 02:05:41,151 INFO [Listener at localhost/35175] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45449 2023-07-15 02:05:41,152 INFO [Listener at localhost/35175] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 02:05:41,153 DEBUG [Listener at localhost/35175] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 02:05:41,154 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:41,156 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:41,158 INFO [Listener at localhost/35175] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45449 connecting to ZooKeeper ensemble=127.0.0.1:55392 2023-07-15 02:05:41,162 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:454490x0, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 02:05:41,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:45449-0x10166ba68a70002 connected 2023-07-15 02:05:41,163 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 02:05:41,164 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 02:05:41,165 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 02:05:41,165 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45449 2023-07-15 02:05:41,166 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45449 2023-07-15 02:05:41,166 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45449 2023-07-15 02:05:41,166 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45449 2023-07-15 02:05:41,167 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45449 2023-07-15 02:05:41,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:05:41,177 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 02:05:41,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:05:41,198 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 02:05:41,198 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 02:05:41,198 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 02:05:41,199 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:05:41,199 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 02:05:41,201 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,45169,1689386739954 from backup master directory 2023-07-15 02:05:41,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 02:05:41,204 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:05:41,204 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 02:05:41,205 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 02:05:41,205 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:05:41,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-15 02:05:41,210 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-15 02:05:41,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/hbase.id with ID: 0ef6a885-3f04-44f7-b450-e45deedfbd77 2023-07-15 02:05:41,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:41,352 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:05:41,395 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x07336fe5 to 127.0.0.1:55392 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 02:05:41,426 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19bee5f4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 02:05:41,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 02:05:41,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-15 02:05:41,471 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-15 02:05:41,471 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-15 02:05:41,473 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 02:05:41,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 02:05:41,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 02:05:41,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/data/master/store-tmp 2023-07-15 02:05:41,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:41,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 02:05:41,541 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 02:05:41,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 02:05:41,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 02:05:41,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 02:05:41,541 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 02:05:41,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 02:05:41,543 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/WALs/jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:05:41,564 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45169%2C1689386739954, suffix=, logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/WALs/jenkins-hbase4.apache.org,45169,1689386739954, archiveDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/oldWALs, maxLogs=10 2023-07-15 02:05:41,617 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK] 2023-07-15 02:05:41,617 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK] 2023-07-15 02:05:41,625 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 02:05:41,688 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/WALs/jenkins-hbase4.apache.org,45169,1689386739954/jenkins-hbase4.apache.org%2C45169%2C1689386739954.1689386741574 2023-07-15 02:05:41,689 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK], DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK]] 2023-07-15 02:05:41,689 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:05:41,689 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:41,693 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 02:05:41,694 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 02:05:41,750 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-15 02:05:41,757 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-15 02:05:41,782 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-15 02:05:41,793 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:41,799 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 02:05:41,801 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 02:05:41,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 02:05:41,819 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 02:05:41,820 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70070947, jitterRate=0.04413847625255585}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 02:05:41,820 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 02:05:41,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-15 02:05:41,841 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-15 02:05:41,841 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-15 02:05:41,843 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-15 02:05:41,845 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-15 02:05:41,877 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-07-15 02:05:41,878 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-15 02:05:41,902 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-15 02:05:41,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-15 02:05:41,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-15 02:05:41,934 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-15 02:05:41,935 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-15 02:05:41,939 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-15 02:05:41,944 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-15 02:05:41,948 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:05:41,949 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-15 02:05:41,949 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-15 02:05:41,961 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-15 02:05:41,965 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 02:05:41,965 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 02:05:41,966 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 02:05:41,966 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:05:41,966 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,45169,1689386739954, sessionid=0x10166ba68a70000, setting cluster-up flag (Was=false) 2023-07-15 02:05:41,982 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:05:41,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-15 02:05:41,989 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:05:41,996 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:05:42,001 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-15 02:05:42,002 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:05:42,005 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.hbase-snapshot/.tmp 2023-07-15 02:05:42,070 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(951): ClusterId : 0ef6a885-3f04-44f7-b450-e45deedfbd77 2023-07-15 02:05:42,073 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(951): ClusterId : 0ef6a885-3f04-44f7-b450-e45deedfbd77 2023-07-15 02:05:42,078 DEBUG [RS:0;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 02:05:42,078 DEBUG [RS:1;jenkins-hbase4:45449] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 02:05:42,086 DEBUG [RS:0;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 02:05:42,086 DEBUG [RS:1;jenkins-hbase4:45449] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 02:05:42,086 DEBUG [RS:0;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 02:05:42,086 DEBUG [RS:1;jenkins-hbase4:45449] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 02:05:42,090 DEBUG [RS:1;jenkins-hbase4:45449] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 02:05:42,090 DEBUG [RS:0;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 02:05:42,091 DEBUG [RS:1;jenkins-hbase4:45449] zookeeper.ReadOnlyZKClient(139): Connect 0x4ffacfae to 127.0.0.1:55392 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 02:05:42,092 DEBUG [RS:0;jenkins-hbase4:39885] zookeeper.ReadOnlyZKClient(139): Connect 0x1ef054b0 to 127.0.0.1:55392 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 02:05:42,098 DEBUG [RS:1;jenkins-hbase4:45449] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@413d2821, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 02:05:42,099 DEBUG [RS:0;jenkins-hbase4:39885] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1037a91e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 02:05:42,099 DEBUG [RS:1;jenkins-hbase4:45449] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4502a4de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 02:05:42,099 DEBUG [RS:0;jenkins-hbase4:39885] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14083fc0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 02:05:42,121 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-15 02:05:42,126 DEBUG [RS:0;jenkins-hbase4:39885] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:39885 2023-07-15 02:05:42,126 DEBUG [RS:1;jenkins-hbase4:45449] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:45449 2023-07-15 02:05:42,130 INFO [RS:0;jenkins-hbase4:39885] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 02:05:42,130 INFO [RS:1;jenkins-hbase4:45449] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 02:05:42,130 INFO [RS:1;jenkins-hbase4:45449] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 02:05:42,130 INFO [RS:0;jenkins-hbase4:39885] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 02:05:42,131 DEBUG [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 02:05:42,131 DEBUG [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 02:05:42,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 02:05:42,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 02:05:42,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 02:05:42,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 02:05:42,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-15 02:05:42,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 02:05:42,133 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,134 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,45169,1689386739954 with isa=jenkins-hbase4.apache.org/172.31.14.131:39885, startcode=1689386741104 2023-07-15 02:05:42,134 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,45169,1689386739954 with isa=jenkins-hbase4.apache.org/172.31.14.131:45449, startcode=1689386741148 2023-07-15 02:05:42,134 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689386772134 2023-07-15 02:05:42,136 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-15 02:05:42,141 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 02:05:42,141 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-15 02:05:42,148 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 02:05:42,148 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-15 02:05:42,154 DEBUG [RS:0;jenkins-hbase4:39885] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 02:05:42,154 DEBUG [RS:1;jenkins-hbase4:45449] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 02:05:42,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-15 02:05:42,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-15 02:05:42,156 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-15 02:05:42,156 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-15 02:05:42,158 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,160 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-15 02:05:42,161 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-15 02:05:42,161 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-15 02:05:42,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-15 02:05:42,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-15 02:05:42,166 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689386742166,5,FailOnTimeoutGroup] 2023-07-15 02:05:42,166 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689386742166,5,FailOnTimeoutGroup] 2023-07-15 02:05:42,166 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,166 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-15 02:05:42,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,197 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 02:05:42,198 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 02:05:42,198 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd 2023-07-15 02:05:42,216 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:42,220 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 02:05:42,223 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info 2023-07-15 02:05:42,224 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 02:05:42,226 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59999, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 02:05:42,226 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:42,226 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58609, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 02:05:42,227 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 02:05:42,229 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/rep_barrier 2023-07-15 02:05:42,230 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 02:05:42,231 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:42,232 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 02:05:42,233 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table 2023-07-15 02:05:42,234 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 02:05:42,235 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:42,237 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740 2023-07-15 02:05:42,238 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740 2023-07-15 02:05:42,239 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45169] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:42,242 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 02:05:42,244 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 02:05:42,249 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 02:05:42,250 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73757736, jitterRate=0.09907591342926025}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 02:05:42,250 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 02:05:42,250 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 02:05:42,251 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 02:05:42,251 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 02:05:42,251 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 02:05:42,251 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 02:05:42,251 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45169] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:42,252 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 02:05:42,253 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 02:05:42,258 DEBUG [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd 2023-07-15 02:05:42,258 DEBUG [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd 2023-07-15 02:05:42,258 DEBUG [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:45773 2023-07-15 02:05:42,258 DEBUG [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:45773 2023-07-15 02:05:42,258 DEBUG [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 02:05:42,258 DEBUG [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 02:05:42,258 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 02:05:42,259 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-15 02:05:42,264 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:05:42,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 02:05:42,269 DEBUG [RS:0;jenkins-hbase4:39885] zookeeper.ZKUtil(162): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:42,269 WARN [RS:0;jenkins-hbase4:39885] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 02:05:42,269 INFO [RS:0;jenkins-hbase4:39885] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 02:05:42,269 DEBUG [RS:1;jenkins-hbase4:45449] zookeeper.ZKUtil(162): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:42,269 DEBUG [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1948): logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:42,270 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,45449,1689386741148] 2023-07-15 02:05:42,269 WARN [RS:1;jenkins-hbase4:45449] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 02:05:42,270 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39885,1689386741104] 2023-07-15 02:05:42,270 INFO [RS:1;jenkins-hbase4:45449] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 02:05:42,270 DEBUG [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1948): logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:42,280 DEBUG [RS:1;jenkins-hbase4:45449] zookeeper.ZKUtil(162): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:42,280 DEBUG [RS:0;jenkins-hbase4:39885] zookeeper.ZKUtil(162): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:42,281 DEBUG [RS:1;jenkins-hbase4:45449] zookeeper.ZKUtil(162): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:42,282 DEBUG [RS:0;jenkins-hbase4:39885] zookeeper.ZKUtil(162): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:42,284 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 02:05:42,286 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-15 02:05:42,293 DEBUG [RS:1;jenkins-hbase4:45449] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 02:05:42,293 DEBUG [RS:0;jenkins-hbase4:39885] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 02:05:42,302 INFO [RS:0;jenkins-hbase4:39885] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 02:05:42,302 INFO [RS:1;jenkins-hbase4:45449] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 02:05:42,322 INFO [RS:0;jenkins-hbase4:39885] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 02:05:42,322 INFO [RS:1;jenkins-hbase4:45449] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 02:05:42,326 INFO [RS:0;jenkins-hbase4:39885] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 02:05:42,326 INFO [RS:1;jenkins-hbase4:45449] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 02:05:42,326 INFO [RS:0;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,326 INFO [RS:1;jenkins-hbase4:45449] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,327 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 02:05:42,327 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 02:05:42,335 INFO [RS:1;jenkins-hbase4:45449] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,335 INFO [RS:0;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,335 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,335 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,336 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,336 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,336 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,336 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,336 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,336 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,336 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,336 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,336 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 02:05:42,336 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 02:05:42,337 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,337 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,337 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,337 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,337 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,337 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,337 DEBUG [RS:1;jenkins-hbase4:45449] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,337 DEBUG [RS:0;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:42,338 INFO [RS:1;jenkins-hbase4:45449] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,338 INFO [RS:0;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,339 INFO [RS:1;jenkins-hbase4:45449] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,339 INFO [RS:0;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,339 INFO [RS:1;jenkins-hbase4:45449] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,339 INFO [RS:0;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,355 INFO [RS:0;jenkins-hbase4:39885] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 02:05:42,355 INFO [RS:1;jenkins-hbase4:45449] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 02:05:42,358 INFO [RS:0;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39885,1689386741104-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,358 INFO [RS:1;jenkins-hbase4:45449] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45449,1689386741148-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,374 INFO [RS:0;jenkins-hbase4:39885] regionserver.Replication(203): jenkins-hbase4.apache.org,39885,1689386741104 started 2023-07-15 02:05:42,374 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,39885,1689386741104, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39885, sessionid=0x10166ba68a70001 2023-07-15 02:05:42,375 DEBUG [RS:0;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 02:05:42,375 DEBUG [RS:0;jenkins-hbase4:39885] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:42,375 DEBUG [RS:0;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39885,1689386741104' 2023-07-15 02:05:42,375 DEBUG [RS:0;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 02:05:42,375 INFO [RS:1;jenkins-hbase4:45449] regionserver.Replication(203): jenkins-hbase4.apache.org,45449,1689386741148 started 2023-07-15 02:05:42,376 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,45449,1689386741148, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:45449, sessionid=0x10166ba68a70002 2023-07-15 02:05:42,376 DEBUG [RS:1;jenkins-hbase4:45449] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 02:05:42,376 DEBUG [RS:1;jenkins-hbase4:45449] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:42,376 DEBUG [RS:0;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 02:05:42,376 DEBUG [RS:1;jenkins-hbase4:45449] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45449,1689386741148' 2023-07-15 02:05:42,377 DEBUG [RS:1;jenkins-hbase4:45449] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 02:05:42,377 DEBUG [RS:0;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 02:05:42,377 DEBUG [RS:1;jenkins-hbase4:45449] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 02:05:42,377 DEBUG [RS:0;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 02:05:42,378 DEBUG [RS:0;jenkins-hbase4:39885] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:42,378 DEBUG [RS:0;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39885,1689386741104' 2023-07-15 02:05:42,378 DEBUG [RS:0;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 02:05:42,378 DEBUG [RS:1;jenkins-hbase4:45449] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 02:05:42,378 DEBUG [RS:1;jenkins-hbase4:45449] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 02:05:42,378 DEBUG [RS:1;jenkins-hbase4:45449] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:42,379 DEBUG [RS:1;jenkins-hbase4:45449] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45449,1689386741148' 2023-07-15 02:05:42,379 DEBUG [RS:1;jenkins-hbase4:45449] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 02:05:42,379 DEBUG [RS:0;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 02:05:42,379 DEBUG [RS:1;jenkins-hbase4:45449] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 02:05:42,379 DEBUG [RS:0;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 02:05:42,380 INFO [RS:0;jenkins-hbase4:39885] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 02:05:42,380 INFO [RS:0;jenkins-hbase4:39885] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 02:05:42,380 DEBUG [RS:1;jenkins-hbase4:45449] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 02:05:42,380 INFO [RS:1;jenkins-hbase4:45449] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 02:05:42,380 INFO [RS:1;jenkins-hbase4:45449] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 02:05:42,439 DEBUG [jenkins-hbase4:45169] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-15 02:05:42,443 DEBUG [jenkins-hbase4:45169] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 02:05:42,449 DEBUG [jenkins-hbase4:45169] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 02:05:42,449 DEBUG [jenkins-hbase4:45169] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 02:05:42,449 DEBUG [jenkins-hbase4:45169] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 02:05:42,452 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45449,1689386741148, state=OPENING 2023-07-15 02:05:42,459 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-15 02:05:42,461 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:05:42,461 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 02:05:42,465 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45449,1689386741148}] 2023-07-15 02:05:42,490 INFO [RS:1;jenkins-hbase4:45449] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45449%2C1689386741148, suffix=, logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148, archiveDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs, maxLogs=32 2023-07-15 02:05:42,490 INFO [RS:0;jenkins-hbase4:39885] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39885%2C1689386741104, suffix=, logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104, archiveDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs, maxLogs=32 2023-07-15 02:05:42,517 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK] 2023-07-15 02:05:42,517 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK] 2023-07-15 02:05:42,517 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK] 2023-07-15 02:05:42,518 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK] 2023-07-15 02:05:42,529 INFO [RS:0;jenkins-hbase4:39885] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493 2023-07-15 02:05:42,529 INFO [RS:1;jenkins-hbase4:45449] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493 2023-07-15 02:05:42,529 DEBUG [RS:0;jenkins-hbase4:39885] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK], DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK]] 2023-07-15 02:05:42,530 DEBUG [RS:1;jenkins-hbase4:45449] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK], DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK]] 2023-07-15 02:05:42,649 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:42,653 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 02:05:42,656 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53510, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 02:05:42,667 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 02:05:42,668 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 02:05:42,672 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta, suffix=.meta, logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148, archiveDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs, maxLogs=32 2023-07-15 02:05:42,689 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK] 2023-07-15 02:05:42,689 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK] 2023-07-15 02:05:42,696 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta 2023-07-15 02:05:42,698 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK], DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK]] 2023-07-15 02:05:42,699 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:05:42,701 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 02:05:42,717 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 02:05:42,756 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 02:05:42,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 02:05:42,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:42,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 02:05:42,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 02:05:42,765 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 02:05:42,767 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info 2023-07-15 02:05:42,767 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info 2023-07-15 02:05:42,767 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 02:05:42,768 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:42,769 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 02:05:42,770 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/rep_barrier 2023-07-15 02:05:42,770 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/rep_barrier 2023-07-15 02:05:42,771 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 02:05:42,772 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:42,772 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 02:05:42,773 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table 2023-07-15 02:05:42,773 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table 2023-07-15 02:05:42,773 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 02:05:42,774 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:42,776 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740 2023-07-15 02:05:42,778 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740 2023-07-15 02:05:42,782 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 02:05:42,784 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 02:05:42,786 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71705568, jitterRate=0.0684962272644043}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 02:05:42,786 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 02:05:42,795 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689386742641 2023-07-15 02:05:42,812 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 02:05:42,813 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 02:05:42,813 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45449,1689386741148, state=OPEN 2023-07-15 02:05:42,816 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 02:05:42,816 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 02:05:42,820 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-15 02:05:42,821 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45449,1689386741148 in 352 msec 2023-07-15 02:05:42,825 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-15 02:05:42,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 554 msec 2023-07-15 02:05:42,831 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 782 msec 2023-07-15 02:05:42,831 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689386742831, completionTime=-1 2023-07-15 02:05:42,831 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-15 02:05:42,832 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-15 02:05:42,898 DEBUG [hconnection-0x7cdb12f2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:05:42,900 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53512, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:05:42,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-15 02:05:42,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689386802916 2023-07-15 02:05:42,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689386862916 2023-07-15 02:05:42,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-07-15 02:05:42,939 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45169,1689386739954-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,939 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45169,1689386739954-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,939 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45169,1689386739954-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,940 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:45169, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,941 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:42,946 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-15 02:05:42,956 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-15 02:05:42,958 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 02:05:42,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-15 02:05:42,971 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 02:05:42,973 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 02:05:42,994 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:42,997 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3 empty. 2023-07-15 02:05:42,998 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:42,998 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-15 02:05:43,026 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-15 02:05:43,029 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => c890783f854dd16d4725d7217b8ccac3, NAME => 'hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp 2023-07-15 02:05:43,045 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:43,045 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing c890783f854dd16d4725d7217b8ccac3, disabling compactions & flushes 2023-07-15 02:05:43,045 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:43,045 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:43,045 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. after waiting 0 ms 2023-07-15 02:05:43,045 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:43,045 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:43,045 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for c890783f854dd16d4725d7217b8ccac3: 2023-07-15 02:05:43,049 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 02:05:43,065 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689386743052"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689386743052"}]},"ts":"1689386743052"} 2023-07-15 02:05:43,097 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 02:05:43,099 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 02:05:43,103 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689386743099"}]},"ts":"1689386743099"} 2023-07-15 02:05:43,107 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-15 02:05:43,111 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 02:05:43,113 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 02:05:43,113 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 02:05:43,113 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 02:05:43,116 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN}] 2023-07-15 02:05:43,119 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN 2023-07-15 02:05:43,120 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45449,1689386741148; forceNewPlan=false, retain=false 2023-07-15 02:05:43,273 INFO [jenkins-hbase4:45169] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 02:05:43,274 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c890783f854dd16d4725d7217b8ccac3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:43,274 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689386743273"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689386743273"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689386743273"}]},"ts":"1689386743273"} 2023-07-15 02:05:43,278 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c890783f854dd16d4725d7217b8ccac3, server=jenkins-hbase4.apache.org,45449,1689386741148}] 2023-07-15 02:05:43,438 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:43,439 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c890783f854dd16d4725d7217b8ccac3, NAME => 'hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:05:43,440 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:43,440 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:43,440 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:43,440 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:43,442 INFO [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:43,445 DEBUG [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/info 2023-07-15 02:05:43,445 DEBUG [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/info 2023-07-15 02:05:43,445 INFO [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c890783f854dd16d4725d7217b8ccac3 columnFamilyName info 2023-07-15 02:05:43,446 INFO [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] regionserver.HStore(310): Store=c890783f854dd16d4725d7217b8ccac3/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:43,448 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:43,448 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:43,453 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:43,456 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 02:05:43,457 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c890783f854dd16d4725d7217b8ccac3; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62183801, jitterRate=-0.07338915765285492}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 02:05:43,457 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c890783f854dd16d4725d7217b8ccac3: 2023-07-15 02:05:43,459 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3., pid=6, masterSystemTime=1689386743431 2023-07-15 02:05:43,463 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:43,463 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:43,465 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c890783f854dd16d4725d7217b8ccac3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:43,465 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689386743464"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689386743464"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689386743464"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689386743464"}]},"ts":"1689386743464"} 2023-07-15 02:05:43,473 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-15 02:05:43,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c890783f854dd16d4725d7217b8ccac3, server=jenkins-hbase4.apache.org,45449,1689386741148 in 192 msec 2023-07-15 02:05:43,477 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-15 02:05:43,478 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN in 357 msec 2023-07-15 02:05:43,479 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 02:05:43,479 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689386743479"}]},"ts":"1689386743479"} 2023-07-15 02:05:43,482 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-15 02:05:43,487 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 02:05:43,489 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 528 msec 2023-07-15 02:05:43,571 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-15 02:05:43,572 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-15 02:05:43,573 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:05:43,607 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-15 02:05:43,625 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 02:05:43,631 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-07-15 02:05:43,640 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-15 02:05:43,651 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 02:05:43,656 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-07-15 02:05:43,666 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-15 02:05:43,670 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-15 02:05:43,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.465sec 2023-07-15 02:05:43,673 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-15 02:05:43,674 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-15 02:05:43,674 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-15 02:05:43,676 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45169,1689386739954-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-15 02:05:43,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45169,1689386739954-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-15 02:05:43,677 DEBUG [Listener at localhost/35175] zookeeper.ReadOnlyZKClient(139): Connect 0x61901cae to 127.0.0.1:55392 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 02:05:43,681 DEBUG [Listener at localhost/35175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c77a3bc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 02:05:43,688 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-15 02:05:43,695 DEBUG [hconnection-0x7bfc2c0d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:05:43,707 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53526, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:05:43,717 INFO [Listener at localhost/35175] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:05:43,729 DEBUG [Listener at localhost/35175] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-15 02:05:43,732 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-15 02:05:43,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 02:05:43,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-15 02:05:43,752 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 02:05:43,754 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 02:05:43,757 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp/data/default/t/f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:43,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-15 02:05:43,758 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp/data/default/t/f58b932419ecf88d64cad61a433c08f6 empty. 2023-07-15 02:05:43,760 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp/data/default/t/f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:43,760 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-15 02:05:43,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 02:05:43,781 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-15 02:05:43,783 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => f58b932419ecf88d64cad61a433c08f6, NAME => 't,,1689386743742.f58b932419ecf88d64cad61a433c08f6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/.tmp 2023-07-15 02:05:43,798 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:43,798 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing f58b932419ecf88d64cad61a433c08f6, disabling compactions & flushes 2023-07-15 02:05:43,798 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:43,798 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:43,798 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. after waiting 0 ms 2023-07-15 02:05:43,798 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:43,798 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:43,798 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for f58b932419ecf88d64cad61a433c08f6: 2023-07-15 02:05:43,803 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 02:05:43,804 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689386743804"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689386743804"}]},"ts":"1689386743804"} 2023-07-15 02:05:43,807 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 02:05:43,808 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 02:05:43,809 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689386743808"}]},"ts":"1689386743808"} 2023-07-15 02:05:43,811 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-15 02:05:43,816 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 02:05:43,818 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 02:05:43,818 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 02:05:43,818 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 02:05:43,818 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN}] 2023-07-15 02:05:43,821 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN 2023-07-15 02:05:43,822 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45449,1689386741148; forceNewPlan=false, retain=false 2023-07-15 02:05:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 02:05:43,972 INFO [jenkins-hbase4:45169] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 02:05:43,973 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f58b932419ecf88d64cad61a433c08f6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:43,974 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689386743973"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689386743973"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689386743973"}]},"ts":"1689386743973"} 2023-07-15 02:05:43,977 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f58b932419ecf88d64cad61a433c08f6, server=jenkins-hbase4.apache.org,45449,1689386741148}] 2023-07-15 02:05:44,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 02:05:44,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f58b932419ecf88d64cad61a433c08f6, NAME => 't,,1689386743742.f58b932419ecf88d64cad61a433c08f6.', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:05:44,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:44,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:44,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:44,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:44,139 INFO [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:44,141 DEBUG [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/f 2023-07-15 02:05:44,141 DEBUG [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/f 2023-07-15 02:05:44,142 INFO [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f58b932419ecf88d64cad61a433c08f6 columnFamilyName f 2023-07-15 02:05:44,143 INFO [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] regionserver.HStore(310): Store=f58b932419ecf88d64cad61a433c08f6/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:44,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:44,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:44,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:44,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 02:05:44,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f58b932419ecf88d64cad61a433c08f6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73371924, jitterRate=0.0933268666267395}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 02:05:44,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f58b932419ecf88d64cad61a433c08f6: 2023-07-15 02:05:44,155 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689386743742.f58b932419ecf88d64cad61a433c08f6., pid=11, masterSystemTime=1689386744130 2023-07-15 02:05:44,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,158 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f58b932419ecf88d64cad61a433c08f6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:44,159 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689386744158"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689386744158"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689386744158"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689386744158"}]},"ts":"1689386744158"} 2023-07-15 02:05:44,164 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-15 02:05:44,164 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f58b932419ecf88d64cad61a433c08f6, server=jenkins-hbase4.apache.org,45449,1689386741148 in 184 msec 2023-07-15 02:05:44,168 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-15 02:05:44,168 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN in 346 msec 2023-07-15 02:05:44,169 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 02:05:44,169 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689386744169"}]},"ts":"1689386744169"} 2023-07-15 02:05:44,171 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-15 02:05:44,174 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 02:05:44,177 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 427 msec 2023-07-15 02:05:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 02:05:44,383 INFO [Listener at localhost/35175] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-15 02:05:44,383 DEBUG [Listener at localhost/35175] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-15 02:05:44,385 INFO [Listener at localhost/35175] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 02:05:44,389 INFO [Listener at localhost/35175] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-15 02:05:44,390 INFO [Listener at localhost/35175] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 02:05:44,390 INFO [Listener at localhost/35175] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-15 02:05:44,440 INFO [Listener at localhost/35175] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=290, OpenFileDescriptor=581, MaxFileDescriptor=60000, SystemLoadAverage=165, ProcessCount=172, AvailableMemoryMB=5509 2023-07-15 02:05:44,442 INFO [Listener at localhost/35175] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-15 02:05:44,446 DEBUG [Listener at localhost/35175] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 02:05:44,447 DEBUG [Listener at localhost/35175] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,447 INFO [Listener at localhost/35175] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-15 02:05:44,447 INFO [Listener at localhost/35175] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-15 02:05:44,450 INFO [Listener at localhost/35175] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689386743479/Put/vlen=2/seqid=0} 2023-07-15 02:05:44,450 INFO [Listener at localhost/35175] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-15 02:05:44,450 INFO [Listener at localhost/35175] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3./info:regioninfo/1689386743464/Put/vlen=41/seqid=0, hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3./info:seqnumDuringOpen/1689386743464/Put/vlen=8/seqid=0, hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3./info:server/1689386743464/Put/vlen=31/seqid=0, hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3./info:serverstartcode/1689386743464/Put/vlen=8/seqid=0, hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3./info:sn/1689386743273/Put/vlen=45/seqid=0, hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3./info:state/1689386743464/Put/vlen=4/seqid=0} 2023-07-15 02:05:44,451 INFO [Listener at localhost/35175] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => c890783f854dd16d4725d7217b8ccac3, NAME => 'hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:05:44,451 INFO [Listener at localhost/35175] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689386744169/Put/vlen=2/seqid=0} 2023-07-15 02:05:44,451 INFO [Listener at localhost/35175] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-15 02:05:44,451 INFO [Listener at localhost/35175] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689386743742.f58b932419ecf88d64cad61a433c08f6./info:regioninfo/1689386744158/Put/vlen=35/seqid=0, t,,1689386743742.f58b932419ecf88d64cad61a433c08f6./info:seqnumDuringOpen/1689386744158/Put/vlen=8/seqid=0, t,,1689386743742.f58b932419ecf88d64cad61a433c08f6./info:server/1689386744158/Put/vlen=31/seqid=0, t,,1689386743742.f58b932419ecf88d64cad61a433c08f6./info:serverstartcode/1689386744158/Put/vlen=8/seqid=0, t,,1689386743742.f58b932419ecf88d64cad61a433c08f6./info:sn/1689386743973/Put/vlen=45/seqid=0, t,,1689386743742.f58b932419ecf88d64cad61a433c08f6./info:state/1689386744158/Put/vlen=4/seqid=0} 2023-07-15 02:05:44,451 INFO [Listener at localhost/35175] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => f58b932419ecf88d64cad61a433c08f6, NAME => 't,,1689386743742.f58b932419ecf88d64cad61a433c08f6.', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:05:44,452 DEBUG [Listener at localhost/35175] zookeeper.ReadOnlyZKClient(139): Connect 0x63046820 to 127.0.0.1:55392 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 02:05:44,457 DEBUG [Listener at localhost/35175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d455998, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 02:05:44,457 INFO [Listener at localhost/35175] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-15 02:05:44,457 INFO [Listener at localhost/35175] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-15 02:05:44,459 DEBUG [hconnection-0x20fe1a29-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:05:44,462 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53538, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:05:44,467 INFO [Listener at localhost/35175] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-15 02:05:44,470 ERROR [Listener at localhost/35175] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,45449,1689386741148: die! ***** 2023-07-15 02:05:44,471 ERROR [Listener at localhost/35175] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 02:05:44,473 DEBUG [Listener at localhost/35175] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 02:05:44,476 DEBUG [Listener at localhost/35175] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 02:05:44,477 DEBUG [Listener at localhost/35175] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 02:05:44,477 DEBUG [Listener at localhost/35175] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 02:05:44,478 INFO [Listener at localhost/35175] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 879230976, "init": 513802240, "max": 2051014656, "used": 394271384 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 111026448 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-15 02:05:44,510 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45169] master.MasterRpcServices(612): jenkins-hbase4.apache.org,45449,1689386741148 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,45449,1689386741148: die! ***** 2023-07-15 02:05:44,516 INFO [Listener at localhost/35175] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,45449,1689386741148' ***** 2023-07-15 02:05:44,517 INFO [Listener at localhost/35175] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-15 02:05:44,517 INFO [RS:1;jenkins-hbase4:45449] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 02:05:44,517 INFO [RS:1;jenkins-hbase4:45449] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 02:05:44,517 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 02:05:44,517 INFO [RS:1;jenkins-hbase4:45449] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 02:05:44,517 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(3305): Received CLOSE for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:44,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45449] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -5697391860377452532 2023-07-15 02:05:44,518 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(3305): Received CLOSE for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:44,519 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:44,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45449] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45449,1689386741148 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 02:05:44,519 DEBUG [RS:1;jenkins-hbase4:45449] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ffacfae to 127.0.0.1:55392 2023-07-15 02:05:44,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f58b932419ecf88d64cad61a433c08f6, disabling compactions & flushes 2023-07-15 02:05:44,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45449] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:53538 deadline: 1689386804518, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45449,1689386741148 aborting 2023-07-15 02:05:44,519 DEBUG [RS:1;jenkins-hbase4:45449] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:05:44,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,520 INFO [RS:1;jenkins-hbase4:45449] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 02:05:44,520 INFO [RS:1;jenkins-hbase4:45449] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 02:05:44,520 INFO [RS:1;jenkins-hbase4:45449] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 02:05:44,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,520 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 02:05:44,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. after waiting 0 ms 2023-07-15 02:05:44,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,520 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 02:05:44,520 DEBUG [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1478): Online Regions={f58b932419ecf88d64cad61a433c08f6=t,,1689386743742.f58b932419ecf88d64cad61a433c08f6., 1588230740=hbase:meta,,1.1588230740, c890783f854dd16d4725d7217b8ccac3=hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.} 2023-07-15 02:05:44,521 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 02:05:44,521 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 02:05:44,521 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,522 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 02:05:44,522 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 02:05:44,522 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 02:05:44,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f58b932419ecf88d64cad61a433c08f6: 2023-07-15 02:05:44,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:44,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c890783f854dd16d4725d7217b8ccac3, disabling compactions & flushes 2023-07-15 02:05:44,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:44,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:44,523 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-15 02:05:44,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. after waiting 0 ms 2023-07-15 02:05:44,524 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 02:05:44,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:44,524 DEBUG [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1504): Waiting on 1588230740, c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:44,524 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:44,524 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 02:05:44,525 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 02:05:44,525 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 02:05:44,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:44,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c890783f854dd16d4725d7217b8ccac3: 2023-07-15 02:05:44,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:44,538 DEBUG [Listener at localhost/35175] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45449,1689386741148 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45449,1689386741148 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-15 02:05:44,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45449] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45449,1689386741148 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 02:05:44,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45449] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:53538 deadline: 1689386804541, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45449,1689386741148 aborting 2023-07-15 02:05:44,645 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45449] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45449,1689386741148 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 02:05:44,645 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45449] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:53538 deadline: 1689386804645, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45449,1689386741148 aborting 2023-07-15 02:05:44,724 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,45449,1689386741148; all regions closed. 2023-07-15 02:05:44,740 DEBUG [RS:1;jenkins-hbase4:45449] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:05:44,740 INFO [RS:1;jenkins-hbase4:45449] regionserver.LeaseManager(133): Closed leases 2023-07-15 02:05:44,741 INFO [RS:1;jenkins-hbase4:45449] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-15 02:05:44,741 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 02:05:44,742 INFO [RS:1;jenkins-hbase4:45449] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45449 2023-07-15 02:05:44,752 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:44,752 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:05:44,752 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:44,752 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:05:44,752 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:05:44,753 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,45449,1689386741148] 2023-07-15 02:05:44,753 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,45449,1689386741148; numProcessing=1 2023-07-15 02:05:44,753 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:45449 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45449 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 02:05:44,754 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:45449 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45449 2023-07-15 02:05:44,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:44,755 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,45449,1689386741148 znode expired, triggering replicatorRemoved event 2023-07-15 02:05:44,755 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,45449,1689386741148 already deleted, retry=false 2023-07-15 02:05:44,755 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,45449,1689386741148 on jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:05:44,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:44,762 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,45449,1689386741148, splitWal=true, meta=true 2023-07-15 02:05:44,763 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,45449,1689386741148 (carryingMeta=true) jenkins-hbase4.apache.org,45449,1689386741148/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@60de68e9[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 02:05:44,765 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45449,1689386741148, splitWal=true, meta=true 2023-07-15 02:05:44,767 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45449,1689386741148, splitWal=true, meta=true, isMeta: true 2023-07-15 02:05:44,769 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting 2023-07-15 02:05:44,770 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,45449,1689386741148 WAL count=1, meta=true 2023-07-15 02:05:44,773 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta}] 2023-07-15 02:05:44,781 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:44,782 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta, worker=jenkins-hbase4.apache.org,39885,1689386741104}] 2023-07-15 02:05:44,854 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 02:05:44,855 INFO [RS:1;jenkins-hbase4:45449] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,45449,1689386741148; zookeeper connection closed. 2023-07-15 02:05:44,855 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:45449-0x10166ba68a70002, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 02:05:44,855 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4333f9c9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4333f9c9 2023-07-15 02:05:44,943 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:44,944 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 02:05:44,946 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 02:05:44,948 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39885] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-15 02:05:44,957 DEBUG [hconnection-0x20fe1a29-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45449 this server is in the failed servers list 2023-07-15 02:05:44,969 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta, size=3.1 K (3185bytes) 2023-07-15 02:05:44,969 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta 2023-07-15 02:05:44,974 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta after 5ms 2023-07-15 02:05:44,979 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta took 12ms 2023-07-15 02:05:44,994 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-15 02:05:44,996 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta so closing down 2023-07-15 02:05:44,996 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 02:05:45,001 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-15 02:05:45,004 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta.temp 2023-07-15 02:05:45,006 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 02:05:45,016 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-15 02:05:45,018 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta.temp to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 02:05:45,018 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 38 ms; skipped=1; WAL=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-15 02:05:45,018 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta, journal: Splitting hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta, size=3.1 K (3185bytes) at 1689386744969Finishing writing output for hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta so closing down at 1689386744996 (+27 ms)Creating recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta.temp at 1689386745004 (+8 ms)3 split writer threads finished at 1689386745006 (+2 ms)Closed recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689386745016 (+10 ms)Rename recovered edits hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta.temp to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689386745018 (+2 ms)Processed 11 edits across 1 Regions in 38 ms; skipped=1; WAL=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689386745018 2023-07-15 02:05:45,019 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta 2023-07-15 02:05:45,021 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-15 02:05:45,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.HMaster(3641): Remote procedure done, pid=14 2023-07-15 02:05:45,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] wal.WALSplitUtil(141): Moved hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs 2023-07-15 02:05:45,033 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-15 02:05:45,034 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta, worker=jenkins-hbase4.apache.org,39885,1689386741104 in 249 msec 2023-07-15 02:05:45,035 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:45,039 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-15 02:05:45,039 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C45449%2C1689386741148.meta.1689386742673.meta, worker=jenkins-hbase4.apache.org,39885,1689386741104 in 264 msec 2023-07-15 02:05:45,041 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting dir is empty, no logs to split. 2023-07-15 02:05:45,042 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,45449,1689386741148 WAL count=0, meta=true 2023-07-15 02:05:45,042 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,45449,1689386741148 WAL splitting is done? wals=0, meta=true 2023-07-15 02:05:45,043 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 02:05:45,044 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 02:05:45,045 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 02:05:45,195 DEBUG [jenkins-hbase4:45169] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 02:05:45,196 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39885,1689386741104, state=OPENING 2023-07-15 02:05:45,197 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 02:05:45,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39885,1689386741104}] 2023-07-15 02:05:45,197 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 02:05:45,260 WARN [ReadOnlyZKClient-127.0.0.1:55392@0x63046820] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-15 02:05:45,261 DEBUG [hconnection-0x20fe1a29-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:05:45,264 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:05:45,265 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39885] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:55168 deadline: 1689386805264, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:45,339 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 02:05:45,356 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 02:05:45,356 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 02:05:45,358 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta, suffix=.meta, logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104, archiveDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs, maxLogs=32 2023-07-15 02:05:45,377 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK] 2023-07-15 02:05:45,378 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK] 2023-07-15 02:05:45,381 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta 2023-07-15 02:05:45,384 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK], DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK]] 2023-07-15 02:05:45,384 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:05:45,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 02:05:45,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 02:05:45,385 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 02:05:45,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 02:05:45,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:45,386 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 02:05:45,386 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 02:05:45,387 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 02:05:45,388 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info 2023-07-15 02:05:45,389 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info 2023-07-15 02:05:45,389 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 02:05:45,390 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:45,390 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 02:05:45,391 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/rep_barrier 2023-07-15 02:05:45,391 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/rep_barrier 2023-07-15 02:05:45,391 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 02:05:45,392 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:45,392 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 02:05:45,394 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table 2023-07-15 02:05:45,394 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table 2023-07-15 02:05:45,394 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 02:05:45,395 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:45,396 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740 2023-07-15 02:05:45,399 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740 2023-07-15 02:05:45,401 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 02:05:45,408 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 02:05:45,409 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-15 02:05:45,459 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/.tmp/info/d39bbc6592ca4257af5246ffe00b112c 2023-07-15 02:05:45,512 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/.tmp/table/c8ba67c71d3741cba283d1eb3e1f7e98 2023-07-15 02:05:45,521 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/.tmp/info/d39bbc6592ca4257af5246ffe00b112c as hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info/d39bbc6592ca4257af5246ffe00b112c 2023-07-15 02:05:45,529 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info/d39bbc6592ca4257af5246ffe00b112c, entries=20, sequenceid=13, filesize=6.9 K 2023-07-15 02:05:45,531 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/.tmp/table/c8ba67c71d3741cba283d1eb3e1f7e98 as hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table/c8ba67c71d3741cba283d1eb3e1f7e98 2023-07-15 02:05:45,540 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table/c8ba67c71d3741cba283d1eb3e1f7e98, entries=4, sequenceid=13, filesize=4.7 K 2023-07-15 02:05:45,540 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 131ms, sequenceid=13, compaction requested=false; wal=null 2023-07-15 02:05:45,542 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-15 02:05:45,545 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 02:05:45,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 02:05:45,550 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 02:05:45,553 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-15 02:05:45,554 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59967606, jitterRate=-0.10641303658485413}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 02:05:45,554 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 02:05:45,556 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689386745350 2023-07-15 02:05:45,558 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 02:05:45,559 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 02:05:45,559 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39885,1689386741104, state=OPEN 2023-07-15 02:05:45,562 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 02:05:45,562 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 02:05:45,564 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-15 02:05:45,564 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39885,1689386741104 in 365 msec 2023-07-15 02:05:45,567 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-15 02:05:45,567 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 521 msec 2023-07-15 02:05:45,567 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,45449,1689386741148 had 3 regions 2023-07-15 02:05:45,568 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45449,1689386741148, splitWal=true, meta=true, isMeta: false 2023-07-15 02:05:45,571 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,45449,1689386741148 WAL count=1, meta=false 2023-07-15 02:05:45,571 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493}] 2023-07-15 02:05:45,573 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:45,574 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493, worker=jenkins-hbase4.apache.org,39885,1689386741104}] 2023-07-15 02:05:45,727 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39885] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-15 02:05:45,741 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493, size=1.6 K (1665bytes) 2023-07-15 02:05:45,741 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493 2023-07-15 02:05:45,741 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493 after 0ms 2023-07-15 02:05:45,745 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493 took 5ms 2023-07-15 02:05:45,748 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for c890783f854dd16d4725d7217b8ccac3: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-15 02:05:45,751 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for f58b932419ecf88d64cad61a433c08f6: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-15 02:05:45,752 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493 so closing down 2023-07-15 02:05:45,752 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 02:05:45,753 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp 2023-07-15 02:05:45,755 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp 2023-07-15 02:05:45,755 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 02:05:45,775 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-15 02:05:45,778 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000013 2023-07-15 02:05:45,793 DEBUG [hconnection-0x20fe1a29-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45449 this server is in the failed servers list 2023-07-15 02:05:46,000 DEBUG [hconnection-0x20fe1a29-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45449 this server is in the failed servers list 2023-07-15 02:05:46,177 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-15 02:05:46,180 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000005 2023-07-15 02:05:46,180 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 435 ms; skipped=2; WAL=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493, size=1.6 K, length=1665, corrupted=false, cancelled=false 2023-07-15 02:05:46,181 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493, journal: Splitting hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493, size=1.6 K (1665bytes) at 1689386745741Finishing writing output for hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493 so closing down at 1689386745752 (+11 ms)Creating recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp at 1689386745753 (+1 ms)Creating recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp at 1689386745755 (+2 ms)3 split writer threads finished at 1689386745755Closed recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689386745775 (+20 ms)Rename recovered edits hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000013 at 1689386745778 (+3 ms)Closed recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689386746178 (+400 ms)Rename recovered edits hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493.temp to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000005 at 1689386746180 (+2 ms)Processed 14 edits across 2 Regions in 435 ms; skipped=2; WAL=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493, size=1.6 K, length=1665, corrupted=false, cancelled=false at 1689386746181 (+1 ms) 2023-07-15 02:05:46,181 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493 2023-07-15 02:05:46,181 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-15 02:05:46,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.HMaster(3641): Remote procedure done, pid=18 2023-07-15 02:05:46,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] wal.WALSplitUtil(141): Moved hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting/jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493 to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs 2023-07-15 02:05:46,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-15 02:05:46,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493, worker=jenkins-hbase4.apache.org,39885,1689386741104 in 613 msec 2023-07-15 02:05:46,192 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:46,195 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-15 02:05:46,195 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C45449%2C1689386741148.1689386742493, worker=jenkins-hbase4.apache.org,39885,1689386741104 in 621 msec 2023-07-15 02:05:46,197 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,45449,1689386741148-splitting dir is empty, no logs to split. 2023-07-15 02:05:46,197 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,45449,1689386741148 WAL count=0, meta=false 2023-07-15 02:05:46,197 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,45449,1689386741148 WAL splitting is done? wals=0, meta=false 2023-07-15 02:05:46,199 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN}, {pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN}] 2023-07-15 02:05:46,200 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN 2023-07-15 02:05:46,201 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN 2023-07-15 02:05:46,201 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 02:05:46,201 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 02:05:46,308 DEBUG [hconnection-0x20fe1a29-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45449 this server is in the failed servers list 2023-07-15 02:05:46,352 DEBUG [jenkins-hbase4:45169] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 02:05:46,352 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=f58b932419ecf88d64cad61a433c08f6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:46,352 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=c890783f854dd16d4725d7217b8ccac3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:46,352 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689386746352"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689386746352"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689386746352"}]},"ts":"1689386746352"} 2023-07-15 02:05:46,352 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689386746352"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689386746352"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689386746352"}]},"ts":"1689386746352"} 2023-07-15 02:05:46,353 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:45449 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45449 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 02:05:46,355 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:45449 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45449 2023-07-15 02:05:46,457 DEBUG [PEWorker-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:05:46,459 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55178, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:05:46,461 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure c890783f854dd16d4725d7217b8ccac3, server=jenkins-hbase4.apache.org,39885,1689386741104}] 2023-07-15 02:05:46,462 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE; OpenRegionProcedure f58b932419ecf88d64cad61a433c08f6, server=jenkins-hbase4.apache.org,39885,1689386741104}] 2023-07-15 02:05:46,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:46,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f58b932419ecf88d64cad61a433c08f6, NAME => 't,,1689386743742.f58b932419ecf88d64cad61a433c08f6.', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:05:46,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:46,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:46,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:46,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:46,621 INFO [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:46,623 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:46,623 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c890783f854dd16d4725d7217b8ccac3, NAME => 'hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:05:46,623 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:46,623 DEBUG [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/f 2023-07-15 02:05:46,623 DEBUG [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/f 2023-07-15 02:05:46,623 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:05:46,624 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:46,624 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:46,624 INFO [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f58b932419ecf88d64cad61a433c08f6 columnFamilyName f 2023-07-15 02:05:46,625 INFO [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] regionserver.HStore(310): Store=f58b932419ecf88d64cad61a433c08f6/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:46,626 INFO [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:46,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:46,627 DEBUG [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/info 2023-07-15 02:05:46,627 DEBUG [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/info 2023-07-15 02:05:46,628 INFO [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c890783f854dd16d4725d7217b8ccac3 columnFamilyName info 2023-07-15 02:05:46,629 INFO [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] regionserver.HStore(310): Store=c890783f854dd16d4725d7217b8ccac3/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:05:46,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:46,632 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:46,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000013 2023-07-15 02:05:46,634 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:46,636 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000005 2023-07-15 02:05:46,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000013 2023-07-15 02:05:46,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f58b932419ecf88d64cad61a433c08f6 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-15 02:05:46,641 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000005 2023-07-15 02:05:46,641 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c890783f854dd16d4725d7217b8ccac3 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-15 02:05:46,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/.tmp/f/102673cc56954f99ab30fdcef15ad2fd 2023-07-15 02:05:46,665 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/.tmp/info/ccdfa71fad1a4aa4b3ee03409bacb7e1 2023-07-15 02:05:46,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/.tmp/f/102673cc56954f99ab30fdcef15ad2fd as hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/f/102673cc56954f99ab30fdcef15ad2fd 2023-07-15 02:05:46,677 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/.tmp/info/ccdfa71fad1a4aa4b3ee03409bacb7e1 as hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/info/ccdfa71fad1a4aa4b3ee03409bacb7e1 2023-07-15 02:05:46,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/f/102673cc56954f99ab30fdcef15ad2fd, entries=10, sequenceid=13, filesize=4.8 K 2023-07-15 02:05:46,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for f58b932419ecf88d64cad61a433c08f6 in 46ms, sequenceid=13, compaction requested=false; wal=null 2023-07-15 02:05:46,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-15 02:05:46,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/0000000000000000013 2023-07-15 02:05:46,686 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/info/ccdfa71fad1a4aa4b3ee03409bacb7e1, entries=2, sequenceid=5, filesize=4.8 K 2023-07-15 02:05:46,686 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c890783f854dd16d4725d7217b8ccac3 in 45ms, sequenceid=5, compaction requested=false; wal=null 2023-07-15 02:05:46,686 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-15 02:05:46,688 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/0000000000000000005 2023-07-15 02:05:46,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:05:46,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-15 02:05:46,696 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f58b932419ecf88d64cad61a433c08f6; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64540920, jitterRate=-0.038265347480773926}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 02:05:46,696 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:05:46,697 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f58b932419ecf88d64cad61a433c08f6: 2023-07-15 02:05:46,698 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689386743742.f58b932419ecf88d64cad61a433c08f6., pid=22, masterSystemTime=1689386746614 2023-07-15 02:05:46,700 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-15 02:05:46,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:46,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:46,701 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c890783f854dd16d4725d7217b8ccac3; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73905407, jitterRate=0.10127638280391693}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 02:05:46,701 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c890783f854dd16d4725d7217b8ccac3: 2023-07-15 02:05:46,702 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=f58b932419ecf88d64cad61a433c08f6, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:46,703 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689386746702"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689386746702"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689386746702"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689386746702"}]},"ts":"1689386746702"} 2023-07-15 02:05:46,703 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3., pid=21, masterSystemTime=1689386746614 2023-07-15 02:05:46,705 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:46,705 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:05:46,706 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=c890783f854dd16d4725d7217b8ccac3, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:46,706 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689386746706"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689386746706"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689386746706"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689386746706"}]},"ts":"1689386746706"} 2023-07-15 02:05:46,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-07-15 02:05:46,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; OpenRegionProcedure f58b932419ecf88d64cad61a433c08f6, server=jenkins-hbase4.apache.org,39885,1689386741104 in 243 msec 2023-07-15 02:05:46,710 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN in 509 msec 2023-07-15 02:05:46,712 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-15 02:05:46,712 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure c890783f854dd16d4725d7217b8ccac3, server=jenkins-hbase4.apache.org,39885,1689386741104 in 247 msec 2023-07-15 02:05:46,714 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=12 2023-07-15 02:05:46,714 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,45449,1689386741148 after splitting done 2023-07-15 02:05:46,714 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN in 513 msec 2023-07-15 02:05:46,714 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase4.apache.org,45449,1689386741148 from processing; numProcessing=0 2023-07-15 02:05:46,717 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,45449,1689386741148, splitWal=true, meta=true in 1.9580 sec 2023-07-15 02:05:46,819 DEBUG [Listener at localhost/35175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63046820 to 127.0.0.1:55392 2023-07-15 02:05:46,819 DEBUG [Listener at localhost/35175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:05:46,819 INFO [Listener at localhost/35175] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-15 02:05:46,833 INFO [Listener at localhost/35175] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=268 (was 290), OpenFileDescriptor=579 (was 581), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=159 (was 165), ProcessCount=171 (was 172), AvailableMemoryMB=5461 (was 5509) 2023-07-15 02:05:46,844 INFO [Listener at localhost/35175] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=268, OpenFileDescriptor=579, MaxFileDescriptor=60000, SystemLoadAverage=159, ProcessCount=171, AvailableMemoryMB=5460 2023-07-15 02:05:46,844 INFO [Listener at localhost/35175] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:45449 2023-07-15 02:05:46,857 INFO [Listener at localhost/35175] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 02:05:46,857 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:46,857 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:46,858 INFO [Listener at localhost/35175] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 02:05:46,858 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 02:05:46,858 INFO [Listener at localhost/35175] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 02:05:46,858 INFO [Listener at localhost/35175] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 02:05:46,860 INFO [Listener at localhost/35175] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33009 2023-07-15 02:05:46,860 INFO [Listener at localhost/35175] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 02:05:46,861 DEBUG [Listener at localhost/35175] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 02:05:46,862 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:46,864 INFO [Listener at localhost/35175] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 02:05:46,865 INFO [Listener at localhost/35175] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33009 connecting to ZooKeeper ensemble=127.0.0.1:55392 2023-07-15 02:05:46,868 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:330090x0, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 02:05:46,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33009-0x10166ba68a70008 connected 2023-07-15 02:05:46,869 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(162): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 02:05:46,870 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(162): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-15 02:05:46,871 DEBUG [Listener at localhost/35175] zookeeper.ZKUtil(164): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 02:05:46,874 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33009 2023-07-15 02:05:46,874 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33009 2023-07-15 02:05:46,875 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33009 2023-07-15 02:05:46,875 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33009 2023-07-15 02:05:46,875 DEBUG [Listener at localhost/35175] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33009 2023-07-15 02:05:46,877 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(951): ClusterId : 0ef6a885-3f04-44f7-b450-e45deedfbd77 2023-07-15 02:05:46,877 DEBUG [RS:2;jenkins-hbase4:33009] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 02:05:46,880 DEBUG [RS:2;jenkins-hbase4:33009] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 02:05:46,880 DEBUG [RS:2;jenkins-hbase4:33009] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 02:05:46,883 DEBUG [RS:2;jenkins-hbase4:33009] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 02:05:46,883 DEBUG [RS:2;jenkins-hbase4:33009] zookeeper.ReadOnlyZKClient(139): Connect 0x7e3e8761 to 127.0.0.1:55392 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 02:05:46,887 DEBUG [RS:2;jenkins-hbase4:33009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70f976d6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 02:05:46,887 DEBUG [RS:2;jenkins-hbase4:33009] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74c6ec11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 02:05:46,895 DEBUG [RS:2;jenkins-hbase4:33009] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:33009 2023-07-15 02:05:46,895 INFO [RS:2;jenkins-hbase4:33009] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 02:05:46,895 INFO [RS:2;jenkins-hbase4:33009] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 02:05:46,895 DEBUG [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 02:05:46,896 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,45169,1689386739954 with isa=jenkins-hbase4.apache.org/172.31.14.131:33009, startcode=1689386746856 2023-07-15 02:05:46,896 DEBUG [RS:2;jenkins-hbase4:33009] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 02:05:46,899 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42741, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 02:05:46,900 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45169] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:05:46,900 DEBUG [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd 2023-07-15 02:05:46,900 DEBUG [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:45773 2023-07-15 02:05:46,900 DEBUG [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 02:05:46,902 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:05:46,902 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:05:46,903 DEBUG [RS:2;jenkins-hbase4:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:05:46,903 WARN [RS:2;jenkins-hbase4:33009] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 02:05:46,903 INFO [RS:2;jenkins-hbase4:33009] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 02:05:46,903 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33009,1689386746856] 2023-07-15 02:05:46,903 DEBUG [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1948): logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:05:46,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:46,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:05:46,908 DEBUG [RS:2;jenkins-hbase4:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:05:46,908 DEBUG [RS:2;jenkins-hbase4:33009] zookeeper.ZKUtil(162): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:05:46,909 DEBUG [RS:2;jenkins-hbase4:33009] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 02:05:46,909 INFO [RS:2;jenkins-hbase4:33009] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 02:05:46,911 INFO [RS:2;jenkins-hbase4:33009] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 02:05:46,911 INFO [RS:2;jenkins-hbase4:33009] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 02:05:46,911 INFO [RS:2;jenkins-hbase4:33009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:46,912 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 02:05:46,913 INFO [RS:2;jenkins-hbase4:33009] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:46,913 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:46,913 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:46,913 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:46,913 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:46,913 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:46,913 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 02:05:46,914 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:46,914 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:46,914 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:46,914 DEBUG [RS:2;jenkins-hbase4:33009] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 02:05:46,915 INFO [RS:2;jenkins-hbase4:33009] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:46,915 INFO [RS:2;jenkins-hbase4:33009] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:46,915 INFO [RS:2;jenkins-hbase4:33009] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:46,926 INFO [RS:2;jenkins-hbase4:33009] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 02:05:46,926 INFO [RS:2;jenkins-hbase4:33009] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33009,1689386746856-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 02:05:46,938 INFO [RS:2;jenkins-hbase4:33009] regionserver.Replication(203): jenkins-hbase4.apache.org,33009,1689386746856 started 2023-07-15 02:05:46,938 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,33009,1689386746856, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33009, sessionid=0x10166ba68a70008 2023-07-15 02:05:46,938 INFO [Listener at localhost/35175] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:33009,5,FailOnTimeoutGroup] 2023-07-15 02:05:46,938 DEBUG [RS:2;jenkins-hbase4:33009] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 02:05:46,938 INFO [Listener at localhost/35175] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:05:46,938 DEBUG [RS:2;jenkins-hbase4:33009] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:05:46,938 INFO [Listener at localhost/35175] client.TestScannerTimeout(177): START ************ test3686b 2023-07-15 02:05:46,938 DEBUG [RS:2;jenkins-hbase4:33009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33009,1689386746856' 2023-07-15 02:05:46,938 DEBUG [RS:2;jenkins-hbase4:33009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 02:05:46,939 DEBUG [RS:2;jenkins-hbase4:33009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 02:05:46,939 DEBUG [RS:2;jenkins-hbase4:33009] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 02:05:46,939 DEBUG [RS:2;jenkins-hbase4:33009] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 02:05:46,939 DEBUG [RS:2;jenkins-hbase4:33009] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:05:46,939 DEBUG [RS:2;jenkins-hbase4:33009] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33009,1689386746856' 2023-07-15 02:05:46,939 DEBUG [RS:2;jenkins-hbase4:33009] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 02:05:46,939 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:45449 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45449 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 02:05:46,940 DEBUG [RS:2;jenkins-hbase4:33009] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 02:05:46,940 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:45449 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45449 2023-07-15 02:05:46,941 DEBUG [RS:2;jenkins-hbase4:33009] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 02:05:46,941 INFO [RS:2;jenkins-hbase4:33009] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 02:05:46,941 INFO [RS:2;jenkins-hbase4:33009] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 02:05:47,043 DEBUG [hconnection-0x7bfc2c0d-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:05:47,045 INFO [RS:2;jenkins-hbase4:33009] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33009%2C1689386746856, suffix=, logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,33009,1689386746856, archiveDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs, maxLogs=32 2023-07-15 02:05:47,047 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:05:47,049 DEBUG [Listener at localhost/35175] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 02:05:47,049 DEBUG [Listener at localhost/35175] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:05:47,050 DEBUG [Listener at localhost/35175] zookeeper.ReadOnlyZKClient(139): Connect 0x52c9d026 to 127.0.0.1:55392 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 02:05:47,058 DEBUG [Listener at localhost/35175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34dd6e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 02:05:47,061 DEBUG [hconnection-0x45d733dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:05:47,066 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:05:47,069 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK] 2023-07-15 02:05:47,070 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK] 2023-07-15 02:05:47,076 INFO [RS:2;jenkins-hbase4:33009] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,33009,1689386746856/jenkins-hbase4.apache.org%2C33009%2C1689386746856.1689386747047 2023-07-15 02:05:47,077 DEBUG [RS:2;jenkins-hbase4:33009] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK], DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK]] 2023-07-15 02:05:48,272 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-15 02:05:50,124 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 02:05:50,125 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-15 02:06:02,077 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 6669549062057885704 expired clientIPAndPort=172.31.14.131:55202, userName=jenkins, regionInfo=t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:04,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.RSRpcServices(3094): Client tried to access missing scanner 6669549062057885704 2023-07-15 02:06:04,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:55202 deadline: 1689386824078, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '6669549062057885704'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-15 02:06:04,080 DEBUG [Listener at localhost/35175] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '6669549062057885704'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '6669549062057885704'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '6669549062057885704'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-15 02:06:04,084 DEBUG [Listener at localhost/35175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52c9d026 to 127.0.0.1:55392 2023-07-15 02:06:04,084 DEBUG [Listener at localhost/35175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:06:04,084 INFO [Listener at localhost/35175] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-15 02:06:04,099 INFO [Listener at localhost/35175] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=296 (was 268) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:55392@0x52c9d026-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: PacketResponder: BP-1740872038-172.31.14.131-1689386736936:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:33009-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:33009 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:45773 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase4:33009Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:55392@0x52c9d026 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/705233044.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:55392@0x7e3e8761-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:55392@0x52c9d026-SendThread(127.0.0.1:55392) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1615619113_17 at /127.0.0.1:54340 [Receiving block BP-1740872038-172.31.14.131-1689386736936:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1740872038-172.31.14.131-1689386736936:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:55392@0x7e3e8761 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/705233044.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:55392@0x7e3e8761-SendThread(127.0.0.1:55392) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd-prefix:jenkins-hbase4.apache.org,33009,1689386746856 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1615619113_17 at /127.0.0.1:53300 [Receiving block BP-1740872038-172.31.14.131-1689386736936:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=33009 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=574 (was 579), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=124 (was 159), ProcessCount=171 (was 171), AvailableMemoryMB=5441 (was 5460) 2023-07-15 02:06:04,112 INFO [Listener at localhost/35175] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=296, OpenFileDescriptor=574, MaxFileDescriptor=60000, SystemLoadAverage=124, ProcessCount=171, AvailableMemoryMB=5441 2023-07-15 02:06:04,113 INFO [Listener at localhost/35175] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:45449 2023-07-15 02:06:04,113 INFO [Listener at localhost/35175] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,45449,1689386741148 2023-07-15 02:06:04,113 INFO [Listener at localhost/35175] client.TestScannerTimeout(104): START************ test2772 2023-07-15 02:06:04,116 DEBUG [Listener at localhost/35175] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 02:06:04,116 DEBUG [Listener at localhost/35175] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:04,117 DEBUG [Listener at localhost/35175] zookeeper.ReadOnlyZKClient(139): Connect 0x775f5d81 to 127.0.0.1:55392 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 02:06:04,121 DEBUG [Listener at localhost/35175] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52db3f80, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 02:06:04,121 ERROR [Listener at localhost/35175] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,39885,1689386741104: die! ***** 2023-07-15 02:06:04,121 ERROR [Listener at localhost/35175] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 02:06:04,121 DEBUG [Listener at localhost/35175] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 02:06:04,122 DEBUG [Listener at localhost/35175] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 02:06:04,128 DEBUG [Listener at localhost/35175] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 02:06:04,130 DEBUG [Listener at localhost/35175] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 02:06:04,150 INFO [Listener at localhost/35175] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 983040000, "init": 513802240, "max": 2051014656, "used": 289185416 }, "NonHeapMemoryUsage": { "committed": 124149760, "init": 2555904, "max": -1, "used": 121603256 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13458, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 65, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 65, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 65, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 65, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 5, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 65, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 17688 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:55392", "tag.serverName": "jenkins-hbase4.apache.org,39885,1689386741104", "tag.clusterId": "0ef6a885-3f04-44f7-b450-e45deedfbd77", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 4, "maxStoreFileCount": 1, "memStoreSize": 1792, "storeFileSize": 21711, "maxStoreFileAge": 16849, "minStoreFileAge": 15642, "avgStoreFileAge": 16233, "numReferenceFiles": 0, "regionServerStartTime": 1689386741104, "averageRegionSize": 7834, "storeFileIndexSize": 1368, "staticIndexSize": 221, "staticBloomSize": 4, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 2492, "localBytesRead": 2492, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819799248, "blockCacheCount": 3, "blockCacheDataBlockCount": 3, "blockCacheSize": 606640, "blockCacheCountHitPercent": 87.5, "blockCacheExpressHitPercent": 87.5, "l1CacheSize": 606640, "l1CacheFreeSize": 819799248, "l1CacheCount": 3, "l1CacheEvictionCount": 0, "l1CacheHitCount": 21, "l1CacheMissCount": 3, "l1CacheHitRatio": 0.875, "l1CacheMissRatio": 0.125, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 8750, "ByteBuffAllocatorPoolAllocationBytes": 931840, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 20, "totalRowActionRequestCount": 24, "readRequestCount": 20, "filteredReadRequestCount": 0, "writeRequestCount": 4, "rpcGetRequestCount": 5, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 11, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 4, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 21, "blockCacheHitCountPrimary": 21, "blockCacheHitCachingCount": 21, "blockCacheMissCount": 3, "blockCacheMissCountPrimary": 3, "blockCacheMissCachingCount": 3, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 3, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 21, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 36, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 2742, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 10, "ScanTime_min": 1, "ScanTime_max": 1, "ScanTime_mean": 1, "ScanTime_25th_percentile": 1, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 4, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 3, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 4, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 21711, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 3, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 10, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 155, "ScanSize_median": 155, "ScanSize_75th_percentile": 653, "ScanSize_90th_percentile": 653, "ScanSize_95th_percentile": 653, "ScanSize_98th_percentile": 653, "ScanSize_99th_percentile": 653, "ScanSize_99.9th_percentile": 653, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 2742, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 3, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.4743477728039096, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.1913526348983332, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-15 02:06:04,151 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45169] master.MasterRpcServices(612): jenkins-hbase4.apache.org,39885,1689386741104 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,39885,1689386741104: die! ***** 2023-07-15 02:06:04,152 INFO [Listener at localhost/35175] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39885,1689386741104' ***** 2023-07-15 02:06:04,153 INFO [Listener at localhost/35175] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-15 02:06:04,153 INFO [RS:0;jenkins-hbase4:39885] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 02:06:04,153 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 02:06:04,153 INFO [RS:0;jenkins-hbase4:39885] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 02:06:04,154 INFO [RS:0;jenkins-hbase4:39885] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 02:06:04,154 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(3305): Received CLOSE for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:04,154 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(3305): Received CLOSE for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:06:04,154 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:06:04,155 DEBUG [RS:0;jenkins-hbase4:39885] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ef054b0 to 127.0.0.1:55392 2023-07-15 02:06:04,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f58b932419ecf88d64cad61a433c08f6, disabling compactions & flushes 2023-07-15 02:06:04,155 DEBUG [RS:0;jenkins-hbase4:39885] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:06:04,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:04,155 INFO [RS:0;jenkins-hbase4:39885] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 02:06:04,155 INFO [RS:0;jenkins-hbase4:39885] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 02:06:04,155 INFO [RS:0;jenkins-hbase4:39885] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 02:06:04,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:04,155 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 02:06:04,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. after waiting 0 ms 2023-07-15 02:06:04,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:04,160 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 02:06:04,161 DEBUG [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1478): Online Regions={f58b932419ecf88d64cad61a433c08f6=t,,1689386743742.f58b932419ecf88d64cad61a433c08f6., 1588230740=hbase:meta,,1.1588230740, c890783f854dd16d4725d7217b8ccac3=hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.} 2023-07-15 02:06:04,161 DEBUG [hconnection-0x49312bf5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:06:04,162 DEBUG [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1504): Waiting on 1588230740, c890783f854dd16d4725d7217b8ccac3, f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:04,162 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 02:06:04,162 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 02:06:04,162 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 02:06:04,162 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 02:06:04,162 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 02:06:04,165 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42630, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:06:04,166 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39885] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39885,1689386741104 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 02:06:04,166 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39885] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:42630 deadline: 1689386824166, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39885,1689386741104 aborting 2023-07-15 02:06:04,173 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 1792 in region hbase:meta,,1.1588230740 2023-07-15 02:06:04,173 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 02:06:04,174 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 02:06:04,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:04,174 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 02:06:04,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f58b932419ecf88d64cad61a433c08f6: 2023-07-15 02:06:04,174 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 02:06:04,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:04,175 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c890783f854dd16d4725d7217b8ccac3, disabling compactions & flushes 2023-07-15 02:06:04,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:04,175 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:04,175 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. after waiting 0 ms 2023-07-15 02:06:04,175 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:04,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:04,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c890783f854dd16d4725d7217b8ccac3: 2023-07-15 02:06:04,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:04,274 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39885] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39885,1689386741104 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 02:06:04,274 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39885] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:42630 deadline: 1689386824274, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39885,1689386741104 aborting 2023-07-15 02:06:04,282 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 02:06:04,339 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-15 02:06:04,339 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-15 02:06:04,362 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,39885,1689386741104; all regions closed. 2023-07-15 02:06:04,384 DEBUG [RS:0;jenkins-hbase4:39885] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:06:04,384 INFO [RS:0;jenkins-hbase4:39885] regionserver.LeaseManager(133): Closed leases 2023-07-15 02:06:04,385 INFO [RS:0;jenkins-hbase4:39885] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-15 02:06:04,385 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 02:06:04,386 INFO [RS:0;jenkins-hbase4:39885] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39885 2023-07-15 02:06:04,394 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:06:04,394 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:06:04,394 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 2023-07-15 02:06:04,394 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:06:04,394 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:06:04,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:04,396 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39885,1689386741104] 2023-07-15 02:06:04,397 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,39885,1689386741104 znode expired, triggering replicatorRemoved event 2023-07-15 02:06:04,397 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39885,1689386741104; numProcessing=1 2023-07-15 02:06:04,412 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39885,1689386741104 already deleted, retry=false 2023-07-15 02:06:04,412 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,39885,1689386741104 on jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:06:04,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:04,414 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=23, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,39885,1689386741104, splitWal=true, meta=true 2023-07-15 02:06:04,414 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=23 for jenkins-hbase4.apache.org,39885,1689386741104 (carryingMeta=true) jenkins-hbase4.apache.org,39885,1689386741104/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@20dc4a34[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 02:06:04,416 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=23, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39885,1689386741104, splitWal=true, meta=true 2023-07-15 02:06:04,418 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39885,1689386741104, splitWal=true, meta=true, isMeta: true 2023-07-15 02:06:04,427 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting 2023-07-15 02:06:04,430 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,39885,1689386741104 WAL count=1, meta=true 2023-07-15 02:06:04,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta}] 2023-07-15 02:06:04,433 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:04,434 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta, worker=jenkins-hbase4.apache.org,33009,1689386746856}] 2023-07-15 02:06:04,477 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39885 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39885 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 02:06:04,479 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39885 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39885 2023-07-15 02:06:04,497 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 02:06:04,497 INFO [RS:0;jenkins-hbase4:39885] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39885,1689386741104; zookeeper connection closed. 2023-07-15 02:06:04,497 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x10166ba68a70001, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 02:06:04,498 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@72201628] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@72201628 2023-07-15 02:06:04,587 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:04,588 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 02:06:04,589 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 02:06:04,589 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33009] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-15 02:06:04,607 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta, size=2.3 K (2403bytes) 2023-07-15 02:06:04,607 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta 2023-07-15 02:06:04,607 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta after 0ms 2023-07-15 02:06:04,612 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta took 6ms 2023-07-15 02:06:04,616 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-15 02:06:04,617 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta so closing down 2023-07-15 02:06:04,617 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 02:06:04,621 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta.temp 2023-07-15 02:06:04,621 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 02:06:04,633 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) 2023-07-15 02:06:04,634 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta.temp to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-15 02:06:04,634 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 5 edits across 1 Regions in 22 ms; skipped=1; WAL=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false 2023-07-15 02:06:04,634 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta, journal: Splitting hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta, size=2.3 K (2403bytes) at 1689386764607Finishing writing output for hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta so closing down at 1689386764617 (+10 ms)Creating recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta.temp at 1689386764621 (+4 ms)3 split writer threads finished at 1689386764622 (+1 ms)Closed recovered edits writer path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) at 1689386764633 (+11 ms)Rename recovered edits hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta.temp to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 at 1689386764634 (+1 ms)Processed 5 edits across 1 Regions in 22 ms; skipped=1; WAL=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false at 1689386764634 2023-07-15 02:06:04,635 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta 2023-07-15 02:06:04,635 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-15 02:06:04,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.HMaster(3641): Remote procedure done, pid=25 2023-07-15 02:06:04,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] wal.WALSplitUtil(141): Moved hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs 2023-07-15 02:06:04,641 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-15 02:06:04,642 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta, worker=jenkins-hbase4.apache.org,33009,1689386746856 in 205 msec 2023-07-15 02:06:04,643 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:04,645 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=23 2023-07-15 02:06:04,645 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C39885%2C1689386741104.meta.1689386745360.meta, worker=jenkins-hbase4.apache.org,33009,1689386746856 in 213 msec 2023-07-15 02:06:04,647 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting dir is empty, no logs to split. 2023-07-15 02:06:04,647 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,39885,1689386741104 WAL count=0, meta=true 2023-07-15 02:06:04,647 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39885,1689386741104 WAL splitting is done? wals=0, meta=true 2023-07-15 02:06:04,648 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 02:06:04,649 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 02:06:04,650 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 02:06:04,780 DEBUG [hconnection-0x49312bf5-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39885 this server is in the failed servers list 2023-07-15 02:06:04,800 DEBUG [jenkins-hbase4:45169] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 02:06:04,801 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33009,1689386746856, state=OPENING 2023-07-15 02:06:04,808 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 02:06:04,808 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 02:06:04,808 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33009,1689386746856}] 2023-07-15 02:06:04,969 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 02:06:04,969 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 02:06:04,971 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33009%2C1689386746856.meta, suffix=.meta, logDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,33009,1689386746856, archiveDir=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs, maxLogs=32 2023-07-15 02:06:04,990 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK] 2023-07-15 02:06:04,991 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK] 2023-07-15 02:06:04,994 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,33009,1689386746856/jenkins-hbase4.apache.org%2C33009%2C1689386746856.meta.1689386764972.meta 2023-07-15 02:06:04,994 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34079,DS-599b7d38-4356-430b-807e-f9689ed79346,DISK], DatanodeInfoWithStorage[127.0.0.1:43351,DS-7e926db4-02dc-46ae-aefd-079bdbd4fbb5,DISK]] 2023-07-15 02:06:04,995 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:06:04,995 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 02:06:04,995 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 02:06:04,995 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 02:06:04,995 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 02:06:04,995 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:06:04,995 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 02:06:04,996 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 02:06:04,997 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 02:06:05,000 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info 2023-07-15 02:06:05,000 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info 2023-07-15 02:06:05,000 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 02:06:05,010 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info/d39bbc6592ca4257af5246ffe00b112c 2023-07-15 02:06:05,011 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:06:05,011 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 02:06:05,013 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/rep_barrier 2023-07-15 02:06:05,013 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/rep_barrier 2023-07-15 02:06:05,015 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 02:06:05,016 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:06:05,016 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 02:06:05,017 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table 2023-07-15 02:06:05,017 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table 2023-07-15 02:06:05,018 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 02:06:05,028 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/table/c8ba67c71d3741cba283d1eb3e1f7e98 2023-07-15 02:06:05,028 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:06:05,029 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740 2023-07-15 02:06:05,031 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740 2023-07-15 02:06:05,032 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-15 02:06:05,039 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 16, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=19, path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-15 02:06:05,039 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-15 02:06:05,053 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=19 (bloomFilter=false), to=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/.tmp/info/7774e0668faa44f19a87bef17b00c75e 2023-07-15 02:06:05,059 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/.tmp/info/7774e0668faa44f19a87bef17b00c75e as hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info/7774e0668faa44f19a87bef17b00c75e 2023-07-15 02:06:05,066 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info/7774e0668faa44f19a87bef17b00c75e, entries=16, sequenceid=19, filesize=6.4 K 2023-07-15 02:06:05,067 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 28ms, sequenceid=19, compaction requested=false; wal=null 2023-07-15 02:06:05,067 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-15 02:06:05,070 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 02:06:05,071 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 02:06:05,074 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=13 2023-07-15 02:06:05,075 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=20; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60098823, jitterRate=-0.10445775091648102}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 02:06:05,075 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 02:06:05,076 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=27, masterSystemTime=1689386764961 2023-07-15 02:06:05,078 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 02:06:05,079 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 02:06:05,079 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33009,1689386746856, state=OPEN 2023-07-15 02:06:05,081 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 02:06:05,081 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 02:06:05,083 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-15 02:06:05,083 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33009,1689386746856 in 273 msec 2023-07-15 02:06:05,085 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=23 2023-07-15 02:06:05,086 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 435 msec 2023-07-15 02:06:05,086 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,39885,1689386741104 had 3 regions 2023-07-15 02:06:05,086 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39885,1689386741104, splitWal=true, meta=true, isMeta: false 2023-07-15 02:06:05,089 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,39885,1689386741104 WAL count=1, meta=false 2023-07-15 02:06:05,089 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493}] 2023-07-15 02:06:05,091 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:05,092 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493, worker=jenkins-hbase4.apache.org,33009,1689386746856}] 2023-07-15 02:06:05,244 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33009] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=29 2023-07-15 02:06:05,257 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493, size=721 (721bytes) 2023-07-15 02:06:05,257 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493 2023-07-15 02:06:05,258 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493 after 1ms 2023-07-15 02:06:05,263 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493 took 6ms 2023-07-15 02:06:05,266 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for f58b932419ecf88d64cad61a433c08f6: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-15 02:06:05,268 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for c890783f854dd16d4725d7217b8ccac3: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-15 02:06:05,268 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493 so closing down 2023-07-15 02:06:05,268 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 02:06:05,268 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 02:06:05,268 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 5 ms; skipped=2; WAL=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493, size=721, length=721, corrupted=false, cancelled=false 2023-07-15 02:06:05,268 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493, journal: Splitting hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493, size=721 (721bytes) at 1689386765257Finishing writing output for hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493 so closing down at 1689386765268 (+11 ms)3 split writer threads finished at 1689386765268Processed 2 edits across 0 Regions in 5 ms; skipped=2; WAL=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493, size=721, length=721, corrupted=false, cancelled=false at 1689386765268 2023-07-15 02:06:05,268 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493 2023-07-15 02:06:05,269 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=29 2023-07-15 02:06:05,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] master.HMaster(3641): Remote procedure done, pid=29 2023-07-15 02:06:05,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45169] wal.WALSplitUtil(141): Moved hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting/jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493 to hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs 2023-07-15 02:06:05,275 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-07-15 02:06:05,275 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493, worker=jenkins-hbase4.apache.org,33009,1689386746856 in 181 msec 2023-07-15 02:06:05,276 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:05,278 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=23 2023-07-15 02:06:05,278 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C39885%2C1689386741104.1689386742493, worker=jenkins-hbase4.apache.org,33009,1689386746856 in 187 msec 2023-07-15 02:06:05,280 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/WALs/jenkins-hbase4.apache.org,39885,1689386741104-splitting dir is empty, no logs to split. 2023-07-15 02:06:05,280 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,39885,1689386741104 WAL count=0, meta=false 2023-07-15 02:06:05,280 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39885,1689386741104 WAL splitting is done? wals=0, meta=false 2023-07-15 02:06:05,282 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN}, {pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN}] 2023-07-15 02:06:05,283 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN 2023-07-15 02:06:05,283 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN 2023-07-15 02:06:05,284 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 02:06:05,284 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 02:06:05,285 DEBUG [hconnection-0x49312bf5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:06:05,288 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53698, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:06:05,294 DEBUG [hconnection-0x49312bf5-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39885 this server is in the failed servers list 2023-07-15 02:06:05,402 DEBUG [hconnection-0x49312bf5-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39885 this server is in the failed servers list 2023-07-15 02:06:05,434 DEBUG [jenkins-hbase4:45169] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 02:06:05,435 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=c890783f854dd16d4725d7217b8ccac3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:05,435 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=f58b932419ecf88d64cad61a433c08f6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:05,435 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689386765434"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689386765434"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689386765434"}]},"ts":"1689386765434"} 2023-07-15 02:06:05,435 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689386765435"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689386765435"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689386765435"}]},"ts":"1689386765435"} 2023-07-15 02:06:05,436 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39885 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39885 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 02:06:05,436 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39885 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39885 2023-07-15 02:06:05,538 DEBUG [PEWorker-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 02:06:05,539 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53704, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 02:06:05,541 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=30, state=RUNNABLE; OpenRegionProcedure f58b932419ecf88d64cad61a433c08f6, server=jenkins-hbase4.apache.org,33009,1689386746856}] 2023-07-15 02:06:05,542 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure c890783f854dd16d4725d7217b8ccac3, server=jenkins-hbase4.apache.org,33009,1689386746856}] 2023-07-15 02:06:05,607 DEBUG [hconnection-0x49312bf5-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39885 this server is in the failed servers list 2023-07-15 02:06:05,698 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:05,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f58b932419ecf88d64cad61a433c08f6, NAME => 't,,1689386743742.f58b932419ecf88d64cad61a433c08f6.', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:06:05,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:05,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:06:05,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:05,699 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:05,700 INFO [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:05,701 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:05,701 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c890783f854dd16d4725d7217b8ccac3, NAME => 'hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.', STARTKEY => '', ENDKEY => ''} 2023-07-15 02:06:05,701 DEBUG [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/f 2023-07-15 02:06:05,701 DEBUG [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/f 2023-07-15 02:06:05,701 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:06:05,701 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 02:06:05,702 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:06:05,702 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:06:05,702 INFO [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f58b932419ecf88d64cad61a433c08f6 columnFamilyName f 2023-07-15 02:06:05,703 INFO [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:06:05,704 DEBUG [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/info 2023-07-15 02:06:05,705 DEBUG [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/info 2023-07-15 02:06:05,705 INFO [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c890783f854dd16d4725d7217b8ccac3 columnFamilyName info 2023-07-15 02:06:05,709 DEBUG [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] regionserver.HStore(539): loaded hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/f/102673cc56954f99ab30fdcef15ad2fd 2023-07-15 02:06:05,710 INFO [StoreOpener-f58b932419ecf88d64cad61a433c08f6-1] regionserver.HStore(310): Store=f58b932419ecf88d64cad61a433c08f6/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:06:05,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:05,713 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:05,713 DEBUG [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] regionserver.HStore(539): loaded hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/info/ccdfa71fad1a4aa4b3ee03409bacb7e1 2023-07-15 02:06:05,713 INFO [StoreOpener-c890783f854dd16d4725d7217b8ccac3-1] regionserver.HStore(310): Store=c890783f854dd16d4725d7217b8ccac3/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 02:06:05,715 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:06:05,716 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:06:05,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:05,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-15 02:06:05,721 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:06:05,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f58b932419ecf88d64cad61a433c08f6; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69125773, jitterRate=0.030054286122322083}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 02:06:05,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f58b932419ecf88d64cad61a433c08f6: 2023-07-15 02:06:05,723 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689386743742.f58b932419ecf88d64cad61a433c08f6., pid=32, masterSystemTime=1689386765694 2023-07-15 02:06:05,724 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-15 02:06:05,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:05,725 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:05,726 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c890783f854dd16d4725d7217b8ccac3; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64819590, jitterRate=-0.0341128408908844}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 02:06:05,726 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c890783f854dd16d4725d7217b8ccac3: 2023-07-15 02:06:05,726 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=f58b932419ecf88d64cad61a433c08f6, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:05,726 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689386743742.f58b932419ecf88d64cad61a433c08f6.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689386765726"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689386765726"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689386765726"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689386765726"}]},"ts":"1689386765726"} 2023-07-15 02:06:05,726 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3., pid=33, masterSystemTime=1689386765694 2023-07-15 02:06:05,728 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:05,728 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:05,729 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=c890783f854dd16d4725d7217b8ccac3, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:05,729 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689386765729"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689386765729"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689386765729"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689386765729"}]},"ts":"1689386765729"} 2023-07-15 02:06:05,730 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=30 2023-07-15 02:06:05,731 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=30, state=SUCCESS; OpenRegionProcedure f58b932419ecf88d64cad61a433c08f6, server=jenkins-hbase4.apache.org,33009,1689386746856 in 187 msec 2023-07-15 02:06:05,732 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=t, region=f58b932419ecf88d64cad61a433c08f6, ASSIGN in 449 msec 2023-07-15 02:06:05,734 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-07-15 02:06:05,734 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure c890783f854dd16d4725d7217b8ccac3, server=jenkins-hbase4.apache.org,33009,1689386746856 in 189 msec 2023-07-15 02:06:05,736 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=23 2023-07-15 02:06:05,736 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,39885,1689386741104 after splitting done 2023-07-15 02:06:05,736 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c890783f854dd16d4725d7217b8ccac3, ASSIGN in 452 msec 2023-07-15 02:06:05,736 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,39885,1689386741104 from processing; numProcessing=0 2023-07-15 02:06:05,738 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,39885,1689386741104, splitWal=true, meta=true in 1.3240 sec 2023-07-15 02:06:05,924 DEBUG [Listener at localhost/35175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x775f5d81 to 127.0.0.1:55392 2023-07-15 02:06:05,924 DEBUG [Listener at localhost/35175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:06:05,924 INFO [Listener at localhost/35175] client.TestScannerTimeout(124): END ************ test2772 2023-07-15 02:06:05,937 INFO [Listener at localhost/35175] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 296), OpenFileDescriptor=582 (was 574) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=122 (was 124), ProcessCount=171 (was 171), AvailableMemoryMB=5435 (was 5441) 2023-07-15 02:06:05,937 INFO [Listener at localhost/35175] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-15 02:06:05,938 INFO [Listener at localhost/35175] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-15 02:06:05,938 DEBUG [Listener at localhost/35175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61901cae to 127.0.0.1:55392 2023-07-15 02:06:05,938 DEBUG [Listener at localhost/35175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:06:05,938 DEBUG [Listener at localhost/35175] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-15 02:06:05,938 DEBUG [Listener at localhost/35175] util.JVMClusterUtil(257): Found active master hash=963843097, stopped=false 2023-07-15 02:06:05,938 INFO [Listener at localhost/35175] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:06:05,942 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 02:06:05,942 INFO [Listener at localhost/35175] procedure2.ProcedureExecutor(629): Stopping 2023-07-15 02:06:05,942 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 02:06:05,942 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:06:05,942 DEBUG [Listener at localhost/35175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07336fe5 to 127.0.0.1:55392 2023-07-15 02:06:05,942 DEBUG [Listener at localhost/35175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:06:05,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 02:06:05,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 02:06:05,943 INFO [Listener at localhost/35175] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,33009,1689386746856' ***** 2023-07-15 02:06:05,943 INFO [Listener at localhost/35175] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-15 02:06:05,943 INFO [RS:2;jenkins-hbase4:33009] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 02:06:05,943 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 02:06:05,943 INFO [RS:2;jenkins-hbase4:33009] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-15 02:06:05,943 INFO [RS:2;jenkins-hbase4:33009] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-15 02:06:05,943 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(3305): Received CLOSE for f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:05,944 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(3305): Received CLOSE for c890783f854dd16d4725d7217b8ccac3 2023-07-15 02:06:05,944 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:05,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f58b932419ecf88d64cad61a433c08f6, disabling compactions & flushes 2023-07-15 02:06:05,944 DEBUG [RS:2;jenkins-hbase4:33009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e3e8761 to 127.0.0.1:55392 2023-07-15 02:06:05,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:05,944 DEBUG [RS:2;jenkins-hbase4:33009] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:06:05,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:05,944 INFO [RS:2;jenkins-hbase4:33009] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 02:06:05,945 INFO [RS:2;jenkins-hbase4:33009] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 02:06:05,945 INFO [RS:2;jenkins-hbase4:33009] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 02:06:05,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. after waiting 0 ms 2023-07-15 02:06:05,945 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 02:06:05,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:05,945 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 02:06:05,945 DEBUG [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1478): Online Regions={f58b932419ecf88d64cad61a433c08f6=t,,1689386743742.f58b932419ecf88d64cad61a433c08f6., 1588230740=hbase:meta,,1.1588230740, c890783f854dd16d4725d7217b8ccac3=hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3.} 2023-07-15 02:06:05,945 DEBUG [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1504): Waiting on 1588230740, c890783f854dd16d4725d7217b8ccac3, f58b932419ecf88d64cad61a433c08f6 2023-07-15 02:06:05,946 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 02:06:05,946 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 02:06:05,946 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 02:06:05,946 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 02:06:05,946 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 02:06:05,947 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-15 02:06:05,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/default/t/f58b932419ecf88d64cad61a433c08f6/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-15 02:06:05,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:05,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f58b932419ecf88d64cad61a433c08f6: 2023-07-15 02:06:05,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689386743742.f58b932419ecf88d64cad61a433c08f6. 2023-07-15 02:06:05,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c890783f854dd16d4725d7217b8ccac3, disabling compactions & flushes 2023-07-15 02:06:05,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:05,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:05,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. after waiting 0 ms 2023-07-15 02:06:05,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:05,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/namespace/c890783f854dd16d4725d7217b8ccac3/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-15 02:06:05,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:05,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c890783f854dd16d4725d7217b8ccac3: 2023-07-15 02:06:05,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689386742957.c890783f854dd16d4725d7217b8ccac3. 2023-07-15 02:06:05,975 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=26 (bloomFilter=false), to=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/.tmp/info/0e0e2a0816ef4300b9a99233cc96f190 2023-07-15 02:06:05,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/.tmp/info/0e0e2a0816ef4300b9a99233cc96f190 as hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info/0e0e2a0816ef4300b9a99233cc96f190 2023-07-15 02:06:05,989 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/info/0e0e2a0816ef4300b9a99233cc96f190, entries=16, sequenceid=26, filesize=6.4 K 2023-07-15 02:06:05,991 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 45ms, sequenceid=26, compaction requested=true 2023-07-15 02:06:06,001 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/data/hbase/meta/1588230740/recovered.edits/29.seqid, newMaxSeqId=29, maxSeqId=19 2023-07-15 02:06:06,002 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 02:06:06,002 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 02:06:06,002 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 02:06:06,002 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 02:06:06,146 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,33009,1689386746856; all regions closed. 2023-07-15 02:06:06,153 DEBUG [RS:2;jenkins-hbase4:33009] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs 2023-07-15 02:06:06,153 INFO [RS:2;jenkins-hbase4:33009] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33009%2C1689386746856.meta:.meta(num 1689386764972) 2023-07-15 02:06:06,159 DEBUG [RS:2;jenkins-hbase4:33009] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/oldWALs 2023-07-15 02:06:06,159 INFO [RS:2;jenkins-hbase4:33009] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33009%2C1689386746856:(num 1689386747047) 2023-07-15 02:06:06,160 DEBUG [RS:2;jenkins-hbase4:33009] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:06:06,160 INFO [RS:2;jenkins-hbase4:33009] regionserver.LeaseManager(133): Closed leases 2023-07-15 02:06:06,160 INFO [RS:2;jenkins-hbase4:33009] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-15 02:06:06,160 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 02:06:06,161 INFO [RS:2;jenkins-hbase4:33009] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33009 2023-07-15 02:06:06,164 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:06:06,164 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33009,1689386746856 2023-07-15 02:06:06,164 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 02:06:06,170 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33009,1689386746856] 2023-07-15 02:06:06,170 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33009,1689386746856; numProcessing=1 2023-07-15 02:06:06,171 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33009,1689386746856 already deleted, retry=false 2023-07-15 02:06:06,171 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33009,1689386746856 expired; onlineServers=0 2023-07-15 02:06:06,171 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,45169,1689386739954' ***** 2023-07-15 02:06:06,171 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-15 02:06:06,172 DEBUG [M:0;jenkins-hbase4:45169] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@784e1e22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 02:06:06,172 INFO [M:0;jenkins-hbase4:45169] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,45169,1689386739954 2023-07-15 02:06:06,172 INFO [M:0;jenkins-hbase4:45169] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,45169,1689386739954; all regions closed. 2023-07-15 02:06:06,172 DEBUG [M:0;jenkins-hbase4:45169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 02:06:06,173 DEBUG [M:0;jenkins-hbase4:45169] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-15 02:06:06,173 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-15 02:06:06,173 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689386742166] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689386742166,5,FailOnTimeoutGroup] 2023-07-15 02:06:06,173 DEBUG [M:0;jenkins-hbase4:45169] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-15 02:06:06,173 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689386742166] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689386742166,5,FailOnTimeoutGroup] 2023-07-15 02:06:06,173 INFO [M:0;jenkins-hbase4:45169] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-15 02:06:06,173 INFO [M:0;jenkins-hbase4:45169] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-15 02:06:06,174 INFO [M:0;jenkins-hbase4:45169] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-15 02:06:06,174 DEBUG [M:0;jenkins-hbase4:45169] master.HMaster(1512): Stopping service threads 2023-07-15 02:06:06,174 INFO [M:0;jenkins-hbase4:45169] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-15 02:06:06,174 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-15 02:06:06,174 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 02:06:06,174 ERROR [M:0;jenkins-hbase4:45169] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1185813668) connection to localhost/127.0.0.1:45773 from jenkins,5,PEWorkerGroup] 2023-07-15 02:06:06,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 02:06:06,175 INFO [M:0;jenkins-hbase4:45169] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-15 02:06:06,175 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-15 02:06:06,175 DEBUG [M:0;jenkins-hbase4:45169] zookeeper.ZKUtil(398): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-15 02:06:06,176 WARN [M:0;jenkins-hbase4:45169] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-15 02:06:06,176 INFO [M:0;jenkins-hbase4:45169] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-15 02:06:06,176 INFO [M:0;jenkins-hbase4:45169] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-15 02:06:06,176 DEBUG [M:0;jenkins-hbase4:45169] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 02:06:06,177 INFO [M:0;jenkins-hbase4:45169] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 02:06:06,177 DEBUG [M:0;jenkins-hbase4:45169] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 02:06:06,177 DEBUG [M:0;jenkins-hbase4:45169] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 02:06:06,177 DEBUG [M:0;jenkins-hbase4:45169] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 02:06:06,177 INFO [M:0;jenkins-hbase4:45169] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=111.30 KB heapSize=134.23 KB 2023-07-15 02:06:06,193 INFO [M:0;jenkins-hbase4:45169] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=111.30 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f1160ee6f7dd4a81ad460a4073a8dce3 2023-07-15 02:06:06,200 DEBUG [M:0;jenkins-hbase4:45169] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f1160ee6f7dd4a81ad460a4073a8dce3 as hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f1160ee6f7dd4a81ad460a4073a8dce3 2023-07-15 02:06:06,206 INFO [M:0;jenkins-hbase4:45169] regionserver.HStore(1080): Added hdfs://localhost:45773/user/jenkins/test-data/12b5fffe-2b0b-a4f7-d86c-3da4bc373dcd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f1160ee6f7dd4a81ad460a4073a8dce3, entries=33, sequenceid=257, filesize=8.5 K 2023-07-15 02:06:06,207 INFO [M:0;jenkins-hbase4:45169] regionserver.HRegion(2948): Finished flush of dataSize ~111.30 KB/113971, heapSize ~134.22 KB/137440, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=257, compaction requested=false 2023-07-15 02:06:06,209 INFO [M:0;jenkins-hbase4:45169] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 02:06:06,209 DEBUG [M:0;jenkins-hbase4:45169] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 02:06:06,213 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 02:06:06,213 INFO [M:0;jenkins-hbase4:45169] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-15 02:06:06,213 INFO [M:0;jenkins-hbase4:45169] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45169 2023-07-15 02:06:06,215 DEBUG [M:0;jenkins-hbase4:45169] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,45169,1689386739954 already deleted, retry=false 2023-07-15 02:06:06,427 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 02:06:06,427 INFO [M:0;jenkins-hbase4:45169] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,45169,1689386739954; zookeeper connection closed. 2023-07-15 02:06:06,427 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): master:45169-0x10166ba68a70000, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 02:06:06,528 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 02:06:06,528 INFO [RS:2;jenkins-hbase4:33009] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,33009,1689386746856; zookeeper connection closed. 2023-07-15 02:06:06,528 DEBUG [Listener at localhost/35175-EventThread] zookeeper.ZKWatcher(600): regionserver:33009-0x10166ba68a70008, quorum=127.0.0.1:55392, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 02:06:06,528 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4fe49b3f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4fe49b3f 2023-07-15 02:06:06,528 INFO [Listener at localhost/35175] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-15 02:06:06,529 WARN [Listener at localhost/35175] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 02:06:06,532 INFO [Listener at localhost/35175] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 02:06:06,637 WARN [BP-1740872038-172.31.14.131-1689386736936 heartbeating to localhost/127.0.0.1:45773] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 02:06:06,637 WARN [BP-1740872038-172.31.14.131-1689386736936 heartbeating to localhost/127.0.0.1:45773] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1740872038-172.31.14.131-1689386736936 (Datanode Uuid ef4d3625-7d61-4442-b5d5-bb90b5c13299) service to localhost/127.0.0.1:45773 2023-07-15 02:06:06,639 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/dfs/data/data3/current/BP-1740872038-172.31.14.131-1689386736936] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 02:06:06,640 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/dfs/data/data4/current/BP-1740872038-172.31.14.131-1689386736936] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 02:06:06,641 WARN [Listener at localhost/35175] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 02:06:06,651 INFO [Listener at localhost/35175] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 02:06:06,756 WARN [BP-1740872038-172.31.14.131-1689386736936 heartbeating to localhost/127.0.0.1:45773] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 02:06:06,756 WARN [BP-1740872038-172.31.14.131-1689386736936 heartbeating to localhost/127.0.0.1:45773] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1740872038-172.31.14.131-1689386736936 (Datanode Uuid 5c995cd3-d4fc-4681-ac43-212a4a8e9da5) service to localhost/127.0.0.1:45773 2023-07-15 02:06:06,757 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/dfs/data/data1/current/BP-1740872038-172.31.14.131-1689386736936] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 02:06:06,757 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/90b686e0-e4f7-190e-f97f-22f222335f9d/cluster_935e168e-7a51-1f61-5660-61eb099d2ae0/dfs/data/data2/current/BP-1740872038-172.31.14.131-1689386736936] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 02:06:06,796 INFO [Listener at localhost/35175] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 02:06:06,908 INFO [Listener at localhost/35175] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-15 02:06:06,917 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 02:06:06,951 INFO [Listener at localhost/35175] hbase.HBaseTestingUtility(1293): Minicluster is down