2023-07-19 12:05:17,611 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-19 12:05:17,852 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642 2023-07-19 12:05:17,866 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-19 12:05:17,867 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5, deleteOnExit=true 2023-07-19 12:05:17,867 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-19 12:05:17,867 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/test.cache.data in system properties and HBase conf 2023-07-19 12:05:17,868 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/hadoop.tmp.dir in system properties and HBase conf 2023-07-19 12:05:17,868 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/hadoop.log.dir in system properties and HBase conf 2023-07-19 12:05:17,869 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-19 12:05:17,869 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-19 12:05:17,869 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-19 12:05:17,977 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-19 12:05:18,353 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-19 12:05:18,358 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-19 12:05:18,359 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-19 12:05:18,359 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-19 12:05:18,359 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-19 12:05:18,360 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-19 12:05:18,360 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-19 12:05:18,361 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-19 12:05:18,361 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-19 12:05:18,361 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-19 12:05:18,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/nfs.dump.dir in system properties and HBase conf 2023-07-19 12:05:18,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/java.io.tmpdir in system properties and HBase conf 2023-07-19 12:05:18,363 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-19 12:05:18,363 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-19 12:05:18,364 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-19 12:05:18,873 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-19 12:05:18,877 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-19 12:05:19,209 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-19 12:05:19,367 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-19 12:05:19,382 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 12:05:19,416 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 12:05:19,448 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/java.io.tmpdir/Jetty_localhost_46831_hdfs____.6xkeiq/webapp 2023-07-19 12:05:19,605 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46831 2023-07-19 12:05:19,619 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-19 12:05:19,619 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-19 12:05:20,028 WARN [Listener at localhost/41359] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 12:05:20,169 WARN [Listener at localhost/41359] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-19 12:05:20,187 WARN [Listener at localhost/41359] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 12:05:20,193 INFO [Listener at localhost/41359] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 12:05:20,197 INFO [Listener at localhost/41359] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/java.io.tmpdir/Jetty_localhost_44533_datanode____.6n6eh0/webapp 2023-07-19 12:05:20,307 INFO [Listener at localhost/41359] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44533 2023-07-19 12:05:20,690 WARN [Listener at localhost/43107] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 12:05:20,756 WARN [Listener at localhost/43107] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-19 12:05:20,760 WARN [Listener at localhost/43107] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 12:05:20,762 INFO [Listener at localhost/43107] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 12:05:20,768 INFO [Listener at localhost/43107] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/java.io.tmpdir/Jetty_localhost_41279_datanode____ne5owu/webapp 2023-07-19 12:05:20,864 INFO [Listener at localhost/43107] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41279 2023-07-19 12:05:20,875 WARN [Listener at localhost/34139] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 12:05:21,179 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8b3dbafc02d6c7d4: Processing first storage report for DS-d3550991-9b4a-4299-ae21-4c1c62fe2160 from datanode e3fc58ad-299d-4544-abc7-36122cc6758c 2023-07-19 12:05:21,181 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8b3dbafc02d6c7d4: from storage DS-d3550991-9b4a-4299-ae21-4c1c62fe2160 node DatanodeRegistration(127.0.0.1:42179, datanodeUuid=e3fc58ad-299d-4544-abc7-36122cc6758c, infoPort=33087, infoSecurePort=0, ipcPort=34139, storageInfo=lv=-57;cid=testClusterID;nsid=1261331287;c=1689768318955), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-19 12:05:21,181 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf2d9aecb202d4fb0: Processing first storage report for DS-eade95c3-5bc2-422c-82de-f340614da657 from datanode a72220e0-0099-491d-b9cc-a632d5171137 2023-07-19 12:05:21,181 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf2d9aecb202d4fb0: from storage DS-eade95c3-5bc2-422c-82de-f340614da657 node DatanodeRegistration(127.0.0.1:45357, datanodeUuid=a72220e0-0099-491d-b9cc-a632d5171137, infoPort=39741, infoSecurePort=0, ipcPort=43107, storageInfo=lv=-57;cid=testClusterID;nsid=1261331287;c=1689768318955), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 12:05:21,181 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8b3dbafc02d6c7d4: Processing first storage report for DS-c48fba36-ee4c-4cfb-81ef-ba1b81381218 from datanode e3fc58ad-299d-4544-abc7-36122cc6758c 2023-07-19 12:05:21,181 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8b3dbafc02d6c7d4: from storage DS-c48fba36-ee4c-4cfb-81ef-ba1b81381218 node DatanodeRegistration(127.0.0.1:42179, datanodeUuid=e3fc58ad-299d-4544-abc7-36122cc6758c, infoPort=33087, infoSecurePort=0, ipcPort=34139, storageInfo=lv=-57;cid=testClusterID;nsid=1261331287;c=1689768318955), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 12:05:21,181 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf2d9aecb202d4fb0: Processing first storage report for DS-b8756545-d1a8-4d08-a001-6c1e49f2a960 from datanode a72220e0-0099-491d-b9cc-a632d5171137 2023-07-19 12:05:21,181 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf2d9aecb202d4fb0: from storage DS-b8756545-d1a8-4d08-a001-6c1e49f2a960 node DatanodeRegistration(127.0.0.1:45357, datanodeUuid=a72220e0-0099-491d-b9cc-a632d5171137, infoPort=39741, infoSecurePort=0, ipcPort=43107, storageInfo=lv=-57;cid=testClusterID;nsid=1261331287;c=1689768318955), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 12:05:21,289 DEBUG [Listener at localhost/34139] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642 2023-07-19 12:05:21,354 INFO [Listener at localhost/34139] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/zookeeper_0, clientPort=53047, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-19 12:05:21,373 INFO [Listener at localhost/34139] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=53047 2023-07-19 12:05:21,385 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:21,388 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:22,055 INFO [Listener at localhost/34139] util.FSUtils(471): Created version file at hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d with version=8 2023-07-19 12:05:22,055 INFO [Listener at localhost/34139] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/hbase-staging 2023-07-19 12:05:22,377 INFO [Listener at localhost/34139] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-19 12:05:22,854 INFO [Listener at localhost/34139] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 12:05:22,886 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:22,886 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:22,886 INFO [Listener at localhost/34139] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 12:05:22,887 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:22,887 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 12:05:23,053 INFO [Listener at localhost/34139] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 12:05:23,133 DEBUG [Listener at localhost/34139] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-19 12:05:23,232 INFO [Listener at localhost/34139] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37041 2023-07-19 12:05:23,243 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:23,245 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:23,270 INFO [Listener at localhost/34139] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37041 connecting to ZooKeeper ensemble=127.0.0.1:53047 2023-07-19 12:05:23,316 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:370410x0, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 12:05:23,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:37041-0x1017d78e43d0000 connected 2023-07-19 12:05:23,350 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 12:05:23,351 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 12:05:23,354 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 12:05:23,367 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37041 2023-07-19 12:05:23,367 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37041 2023-07-19 12:05:23,367 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37041 2023-07-19 12:05:23,368 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37041 2023-07-19 12:05:23,369 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37041 2023-07-19 12:05:23,375 INFO [Listener at localhost/34139] master.HMaster(444): hbase.rootdir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d, hbase.cluster.distributed=false 2023-07-19 12:05:23,446 INFO [Listener at localhost/34139] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 12:05:23,447 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:23,447 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:23,447 INFO [Listener at localhost/34139] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 12:05:23,447 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:23,447 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 12:05:23,453 INFO [Listener at localhost/34139] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 12:05:23,455 INFO [Listener at localhost/34139] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39213 2023-07-19 12:05:23,457 INFO [Listener at localhost/34139] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 12:05:23,464 DEBUG [Listener at localhost/34139] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 12:05:23,465 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:23,467 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:23,469 INFO [Listener at localhost/34139] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39213 connecting to ZooKeeper ensemble=127.0.0.1:53047 2023-07-19 12:05:23,473 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:392130x0, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 12:05:23,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39213-0x1017d78e43d0001 connected 2023-07-19 12:05:23,474 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 12:05:23,475 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 12:05:23,476 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 12:05:23,477 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39213 2023-07-19 12:05:23,477 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39213 2023-07-19 12:05:23,477 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39213 2023-07-19 12:05:23,478 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39213 2023-07-19 12:05:23,478 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39213 2023-07-19 12:05:23,492 INFO [Listener at localhost/34139] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 12:05:23,492 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:23,492 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:23,492 INFO [Listener at localhost/34139] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 12:05:23,492 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:23,493 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 12:05:23,493 INFO [Listener at localhost/34139] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 12:05:23,494 INFO [Listener at localhost/34139] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35959 2023-07-19 12:05:23,495 INFO [Listener at localhost/34139] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 12:05:23,495 DEBUG [Listener at localhost/34139] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 12:05:23,496 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:23,498 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:23,500 INFO [Listener at localhost/34139] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35959 connecting to ZooKeeper ensemble=127.0.0.1:53047 2023-07-19 12:05:23,504 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:359590x0, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 12:05:23,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:35959-0x1017d78e43d0002 connected 2023-07-19 12:05:23,505 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 12:05:23,506 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 12:05:23,507 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 12:05:23,507 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35959 2023-07-19 12:05:23,507 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35959 2023-07-19 12:05:23,508 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35959 2023-07-19 12:05:23,508 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35959 2023-07-19 12:05:23,509 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35959 2023-07-19 12:05:23,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:23,520 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-19 12:05:23,521 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:23,540 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 12:05:23,540 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 12:05:23,540 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 12:05:23,540 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:23,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 12:05:23,542 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,37041,1689768322202 from backup master directory 2023-07-19 12:05:23,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 12:05:23,546 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:23,546 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-19 12:05:23,547 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 12:05:23,547 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:23,549 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-19 12:05:23,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-19 12:05:23,633 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/hbase.id with ID: fc90d035-6952-4500-aaae-cb5f010aeb03 2023-07-19 12:05:23,673 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:23,688 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:23,748 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x4b3841ff to 127.0.0.1:53047 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 12:05:23,794 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77e9676d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 12:05:23,817 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-19 12:05:23,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-19 12:05:23,836 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-19 12:05:23,836 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-19 12:05:23,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-19 12:05:23,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-19 12:05:23,844 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 12:05:23,877 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/data/master/store-tmp 2023-07-19 12:05:23,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:23,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-19 12:05:23,910 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 12:05:23,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 12:05:23,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-19 12:05:23,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 12:05:23,910 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 12:05:23,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 12:05:23,912 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/WALs/jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:23,932 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37041%2C1689768322202, suffix=, logDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/WALs/jenkins-hbase4.apache.org,37041,1689768322202, archiveDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/oldWALs, maxLogs=10 2023-07-19 12:05:23,988 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK] 2023-07-19 12:05:23,988 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK] 2023-07-19 12:05:23,996 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 12:05:24,083 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/WALs/jenkins-hbase4.apache.org,37041,1689768322202/jenkins-hbase4.apache.org%2C37041%2C1689768322202.1689768323942 2023-07-19 12:05:24,084 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK], DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK]] 2023-07-19 12:05:24,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:24,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:24,095 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 12:05:24,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 12:05:24,166 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-19 12:05:24,175 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-19 12:05:24,209 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-19 12:05:24,223 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:24,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-19 12:05:24,233 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-19 12:05:24,255 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 12:05:24,269 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 12:05:24,271 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63316149, jitterRate=-0.056515857577323914}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 12:05:24,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 12:05:24,273 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-19 12:05:24,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-19 12:05:24,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-19 12:05:24,314 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-19 12:05:24,316 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-19 12:05:24,362 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 45 msec 2023-07-19 12:05:24,362 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-19 12:05:24,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-19 12:05:24,406 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-19 12:05:24,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-19 12:05:24,447 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-19 12:05:24,448 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-19 12:05:24,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-19 12:05:24,461 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-19 12:05:24,465 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:24,467 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-19 12:05:24,468 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-19 12:05:24,484 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-19 12:05:24,489 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 12:05:24,489 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 12:05:24,490 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 12:05:24,490 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:24,491 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,37041,1689768322202, sessionid=0x1017d78e43d0000, setting cluster-up flag (Was=false) 2023-07-19 12:05:24,511 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:24,520 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-19 12:05:24,524 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:24,530 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:24,537 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-19 12:05:24,542 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:24,546 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.hbase-snapshot/.tmp 2023-07-19 12:05:24,612 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(951): ClusterId : fc90d035-6952-4500-aaae-cb5f010aeb03 2023-07-19 12:05:24,613 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(951): ClusterId : fc90d035-6952-4500-aaae-cb5f010aeb03 2023-07-19 12:05:24,618 DEBUG [RS:1;jenkins-hbase4:35959] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 12:05:24,618 DEBUG [RS:0;jenkins-hbase4:39213] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 12:05:24,625 DEBUG [RS:1;jenkins-hbase4:35959] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 12:05:24,625 DEBUG [RS:0;jenkins-hbase4:39213] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 12:05:24,625 DEBUG [RS:1;jenkins-hbase4:35959] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 12:05:24,625 DEBUG [RS:0;jenkins-hbase4:39213] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 12:05:24,630 DEBUG [RS:1;jenkins-hbase4:35959] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 12:05:24,630 DEBUG [RS:0;jenkins-hbase4:39213] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 12:05:24,632 DEBUG [RS:0;jenkins-hbase4:39213] zookeeper.ReadOnlyZKClient(139): Connect 0x5bf435c3 to 127.0.0.1:53047 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 12:05:24,632 DEBUG [RS:1;jenkins-hbase4:35959] zookeeper.ReadOnlyZKClient(139): Connect 0x1dd9a9b6 to 127.0.0.1:53047 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 12:05:24,637 DEBUG [RS:1;jenkins-hbase4:35959] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@131e447d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 12:05:24,637 DEBUG [RS:0;jenkins-hbase4:39213] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5533c0cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 12:05:24,638 DEBUG [RS:1;jenkins-hbase4:35959] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1721c2c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 12:05:24,638 DEBUG [RS:0;jenkins-hbase4:39213] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@510c5eb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 12:05:24,667 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:35959 2023-07-19 12:05:24,668 DEBUG [RS:0;jenkins-hbase4:39213] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:39213 2023-07-19 12:05:24,672 INFO [RS:0;jenkins-hbase4:39213] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 12:05:24,673 INFO [RS:0;jenkins-hbase4:39213] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 12:05:24,673 DEBUG [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 12:05:24,672 INFO [RS:1;jenkins-hbase4:35959] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 12:05:24,673 INFO [RS:1;jenkins-hbase4:35959] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 12:05:24,673 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 12:05:24,676 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,37041,1689768322202 with isa=jenkins-hbase4.apache.org/172.31.14.131:35959, startcode=1689768323491 2023-07-19 12:05:24,676 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,37041,1689768322202 with isa=jenkins-hbase4.apache.org/172.31.14.131:39213, startcode=1689768323446 2023-07-19 12:05:24,699 DEBUG [RS:0;jenkins-hbase4:39213] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 12:05:24,699 DEBUG [RS:1;jenkins-hbase4:35959] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 12:05:24,713 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-19 12:05:24,728 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 12:05:24,728 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 12:05:24,728 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 12:05:24,729 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 12:05:24,729 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-19 12:05:24,729 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,729 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 12:05:24,729 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,735 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689768354735 2023-07-19 12:05:24,737 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-19 12:05:24,741 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-19 12:05:24,741 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-19 12:05:24,748 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-19 12:05:24,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-19 12:05:24,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-19 12:05:24,756 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-19 12:05:24,756 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-19 12:05:24,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-19 12:05:24,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,758 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50555, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 12:05:24,758 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60511, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 12:05:24,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-19 12:05:24,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-19 12:05:24,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-19 12:05:24,763 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-19 12:05:24,764 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-19 12:05:24,766 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689768324765,5,FailOnTimeoutGroup] 2023-07-19 12:05:24,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689768324767,5,FailOnTimeoutGroup] 2023-07-19 12:05:24,767 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,767 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-19 12:05:24,768 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37041] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 12:05:24,771 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,771 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,791 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-19 12:05:24,792 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37041] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:24,793 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-19 12:05:24,793 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d 2023-07-19 12:05:24,800 DEBUG [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d 2023-07-19 12:05:24,800 DEBUG [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:41359 2023-07-19 12:05:24,800 DEBUG [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 12:05:24,809 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:24,811 DEBUG [RS:0;jenkins-hbase4:39213] zookeeper.ZKUtil(162): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:24,812 WARN [RS:0;jenkins-hbase4:39213] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 12:05:24,812 INFO [RS:0;jenkins-hbase4:39213] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 12:05:24,813 DEBUG [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1948): logDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:24,815 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39213,1689768323446] 2023-07-19 12:05:24,818 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:24,819 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(2830): Master is not running yet 2023-07-19 12:05:24,820 WARN [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-19 12:05:24,821 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 12:05:24,823 DEBUG [RS:0;jenkins-hbase4:39213] zookeeper.ZKUtil(162): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:24,823 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/info 2023-07-19 12:05:24,824 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 12:05:24,825 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:24,825 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 12:05:24,828 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/rep_barrier 2023-07-19 12:05:24,828 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 12:05:24,829 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:24,830 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 12:05:24,832 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/table 2023-07-19 12:05:24,832 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 12:05:24,833 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:24,834 DEBUG [RS:0;jenkins-hbase4:39213] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 12:05:24,835 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740 2023-07-19 12:05:24,836 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740 2023-07-19 12:05:24,840 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 12:05:24,842 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 12:05:24,842 INFO [RS:0;jenkins-hbase4:39213] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 12:05:24,846 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 12:05:24,847 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71401364, jitterRate=0.06396323442459106}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 12:05:24,847 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 12:05:24,847 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 12:05:24,847 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 12:05:24,847 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 12:05:24,847 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 12:05:24,847 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 12:05:24,848 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 12:05:24,848 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 12:05:24,853 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-19 12:05:24,853 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-19 12:05:24,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-19 12:05:24,865 INFO [RS:0;jenkins-hbase4:39213] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 12:05:24,869 INFO [RS:0;jenkins-hbase4:39213] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 12:05:24,869 INFO [RS:0;jenkins-hbase4:39213] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,870 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 12:05:24,873 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-19 12:05:24,875 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-19 12:05:24,878 INFO [RS:0;jenkins-hbase4:39213] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,879 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,879 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,879 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,879 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,879 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,879 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 12:05:24,879 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,880 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,880 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,880 DEBUG [RS:0;jenkins-hbase4:39213] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,881 INFO [RS:0;jenkins-hbase4:39213] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,881 INFO [RS:0;jenkins-hbase4:39213] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,881 INFO [RS:0;jenkins-hbase4:39213] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,898 INFO [RS:0;jenkins-hbase4:39213] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 12:05:24,900 INFO [RS:0;jenkins-hbase4:39213] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39213,1689768323446-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,917 INFO [RS:0;jenkins-hbase4:39213] regionserver.Replication(203): jenkins-hbase4.apache.org,39213,1689768323446 started 2023-07-19 12:05:24,917 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,39213,1689768323446, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39213, sessionid=0x1017d78e43d0001 2023-07-19 12:05:24,918 DEBUG [RS:0;jenkins-hbase4:39213] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 12:05:24,918 DEBUG [RS:0;jenkins-hbase4:39213] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:24,918 DEBUG [RS:0;jenkins-hbase4:39213] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39213,1689768323446' 2023-07-19 12:05:24,918 DEBUG [RS:0;jenkins-hbase4:39213] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 12:05:24,919 DEBUG [RS:0;jenkins-hbase4:39213] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 12:05:24,919 DEBUG [RS:0;jenkins-hbase4:39213] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 12:05:24,920 DEBUG [RS:0;jenkins-hbase4:39213] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 12:05:24,920 DEBUG [RS:0;jenkins-hbase4:39213] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:24,920 DEBUG [RS:0;jenkins-hbase4:39213] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39213,1689768323446' 2023-07-19 12:05:24,920 DEBUG [RS:0;jenkins-hbase4:39213] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 12:05:24,920 DEBUG [RS:0;jenkins-hbase4:39213] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 12:05:24,920 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,37041,1689768322202 with isa=jenkins-hbase4.apache.org/172.31.14.131:35959, startcode=1689768323491 2023-07-19 12:05:24,921 DEBUG [RS:0;jenkins-hbase4:39213] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 12:05:24,921 INFO [RS:0;jenkins-hbase4:39213] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 12:05:24,921 INFO [RS:0;jenkins-hbase4:39213] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 12:05:24,922 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37041] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:24,924 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d 2023-07-19 12:05:24,924 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:41359 2023-07-19 12:05:24,924 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 12:05:24,932 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:24,932 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:24,933 DEBUG [RS:1;jenkins-hbase4:35959] zookeeper.ZKUtil(162): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:24,933 WARN [RS:1;jenkins-hbase4:35959] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 12:05:24,933 INFO [RS:1;jenkins-hbase4:35959] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 12:05:24,933 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35959,1689768323491] 2023-07-19 12:05:24,933 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1948): logDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:24,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:24,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:24,939 DEBUG [RS:1;jenkins-hbase4:35959] zookeeper.ZKUtil(162): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:24,940 DEBUG [RS:1;jenkins-hbase4:35959] zookeeper.ZKUtil(162): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:24,941 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 12:05:24,942 INFO [RS:1;jenkins-hbase4:35959] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 12:05:24,949 INFO [RS:1;jenkins-hbase4:35959] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 12:05:24,950 INFO [RS:1;jenkins-hbase4:35959] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 12:05:24,950 INFO [RS:1;jenkins-hbase4:35959] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,950 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 12:05:24,952 INFO [RS:1;jenkins-hbase4:35959] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,952 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,953 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,953 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,953 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,953 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,953 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 12:05:24,953 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,953 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,953 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,953 DEBUG [RS:1;jenkins-hbase4:35959] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:24,957 INFO [RS:1;jenkins-hbase4:35959] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,957 INFO [RS:1;jenkins-hbase4:35959] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,957 INFO [RS:1;jenkins-hbase4:35959] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,968 INFO [RS:1;jenkins-hbase4:35959] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 12:05:24,969 INFO [RS:1;jenkins-hbase4:35959] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35959,1689768323491-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:24,980 INFO [RS:1;jenkins-hbase4:35959] regionserver.Replication(203): jenkins-hbase4.apache.org,35959,1689768323491 started 2023-07-19 12:05:24,980 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,35959,1689768323491, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35959, sessionid=0x1017d78e43d0002 2023-07-19 12:05:24,980 DEBUG [RS:1;jenkins-hbase4:35959] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 12:05:24,980 DEBUG [RS:1;jenkins-hbase4:35959] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:24,981 DEBUG [RS:1;jenkins-hbase4:35959] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35959,1689768323491' 2023-07-19 12:05:24,981 DEBUG [RS:1;jenkins-hbase4:35959] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 12:05:24,982 DEBUG [RS:1;jenkins-hbase4:35959] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 12:05:24,982 DEBUG [RS:1;jenkins-hbase4:35959] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 12:05:24,982 DEBUG [RS:1;jenkins-hbase4:35959] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 12:05:24,982 DEBUG [RS:1;jenkins-hbase4:35959] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:24,982 DEBUG [RS:1;jenkins-hbase4:35959] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35959,1689768323491' 2023-07-19 12:05:24,982 DEBUG [RS:1;jenkins-hbase4:35959] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 12:05:24,983 DEBUG [RS:1;jenkins-hbase4:35959] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 12:05:24,983 DEBUG [RS:1;jenkins-hbase4:35959] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 12:05:24,983 INFO [RS:1;jenkins-hbase4:35959] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 12:05:24,983 INFO [RS:1;jenkins-hbase4:35959] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 12:05:25,027 DEBUG [jenkins-hbase4:37041] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-19 12:05:25,031 DEBUG [jenkins-hbase4:37041] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 12:05:25,032 INFO [RS:0;jenkins-hbase4:39213] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39213%2C1689768323446, suffix=, logDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446, archiveDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs, maxLogs=32 2023-07-19 12:05:25,036 DEBUG [jenkins-hbase4:37041] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 12:05:25,036 DEBUG [jenkins-hbase4:37041] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 12:05:25,036 DEBUG [jenkins-hbase4:37041] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 12:05:25,039 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39213,1689768323446, state=OPENING 2023-07-19 12:05:25,046 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-19 12:05:25,047 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:25,048 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 12:05:25,052 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK] 2023-07-19 12:05:25,053 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK] 2023-07-19 12:05:25,054 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39213,1689768323446}] 2023-07-19 12:05:25,058 INFO [RS:0;jenkins-hbase4:39213] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034 2023-07-19 12:05:25,058 DEBUG [RS:0;jenkins-hbase4:39213] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK], DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK]] 2023-07-19 12:05:25,087 INFO [RS:1;jenkins-hbase4:35959] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35959%2C1689768323491, suffix=, logDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491, archiveDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs, maxLogs=32 2023-07-19 12:05:25,105 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK] 2023-07-19 12:05:25,106 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK] 2023-07-19 12:05:25,111 INFO [RS:1;jenkins-hbase4:35959] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088 2023-07-19 12:05:25,111 DEBUG [RS:1;jenkins-hbase4:35959] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK], DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK]] 2023-07-19 12:05:25,239 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:25,242 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 12:05:25,245 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 12:05:25,258 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-19 12:05:25,258 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 12:05:25,261 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta, suffix=.meta, logDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446, archiveDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs, maxLogs=32 2023-07-19 12:05:25,278 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK] 2023-07-19 12:05:25,279 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK] 2023-07-19 12:05:25,284 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta 2023-07-19 12:05:25,284 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK], DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK]] 2023-07-19 12:05:25,285 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:25,288 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-19 12:05:25,311 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-19 12:05:25,351 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-19 12:05:25,356 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-19 12:05:25,357 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:25,357 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-19 12:05:25,357 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-19 12:05:25,360 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 12:05:25,363 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/info 2023-07-19 12:05:25,364 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/info 2023-07-19 12:05:25,364 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 12:05:25,366 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:25,366 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 12:05:25,368 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/rep_barrier 2023-07-19 12:05:25,368 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/rep_barrier 2023-07-19 12:05:25,368 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 12:05:25,369 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:25,370 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 12:05:25,371 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/table 2023-07-19 12:05:25,371 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/table 2023-07-19 12:05:25,372 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 12:05:25,373 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:25,375 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740 2023-07-19 12:05:25,377 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740 2023-07-19 12:05:25,381 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 12:05:25,384 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 12:05:25,386 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62926360, jitterRate=-0.0623241662979126}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 12:05:25,386 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 12:05:25,396 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689768325232 2023-07-19 12:05:25,417 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-19 12:05:25,418 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-19 12:05:25,419 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39213,1689768323446, state=OPEN 2023-07-19 12:05:25,423 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 12:05:25,423 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 12:05:25,427 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-19 12:05:25,427 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39213,1689768323446 in 369 msec 2023-07-19 12:05:25,432 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-19 12:05:25,432 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 567 msec 2023-07-19 12:05:25,437 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 823 msec 2023-07-19 12:05:25,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689768325437, completionTime=-1 2023-07-19 12:05:25,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-19 12:05:25,438 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-19 12:05:25,508 DEBUG [hconnection-0x38877344-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:25,511 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:25,527 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-19 12:05:25,527 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689768385527 2023-07-19 12:05:25,528 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689768445528 2023-07-19 12:05:25,528 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 90 msec 2023-07-19 12:05:25,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37041,1689768322202-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:25,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37041,1689768322202-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:25,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37041,1689768322202-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:25,552 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:37041, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:25,552 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:25,558 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-19 12:05:25,566 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-19 12:05:25,567 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-19 12:05:25,577 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-19 12:05:25,579 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-19 12:05:25,582 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-19 12:05:25,603 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:25,605 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738 empty. 2023-07-19 12:05:25,607 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:25,607 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-19 12:05:25,636 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-19 12:05:25,638 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7b841029b07ea6e492c2c28c4a93a738, NAME => 'hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp 2023-07-19 12:05:25,654 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:25,655 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 7b841029b07ea6e492c2c28c4a93a738, disabling compactions & flushes 2023-07-19 12:05:25,655 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:25,655 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:25,655 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. after waiting 0 ms 2023-07-19 12:05:25,655 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:25,655 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:25,655 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 7b841029b07ea6e492c2c28c4a93a738: 2023-07-19 12:05:25,659 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-19 12:05:25,674 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689768325662"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689768325662"}]},"ts":"1689768325662"} 2023-07-19 12:05:25,701 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-19 12:05:25,703 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-19 12:05:25,707 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689768325703"}]},"ts":"1689768325703"} 2023-07-19 12:05:25,712 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-19 12:05:25,718 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 12:05:25,719 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 12:05:25,719 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 12:05:25,719 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 12:05:25,721 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN}] 2023-07-19 12:05:25,724 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN 2023-07-19 12:05:25,725 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35959,1689768323491; forceNewPlan=false, retain=false 2023-07-19 12:05:25,879 INFO [jenkins-hbase4:37041] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-19 12:05:25,880 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7b841029b07ea6e492c2c28c4a93a738, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:25,880 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689768325880"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689768325880"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689768325880"}]},"ts":"1689768325880"} 2023-07-19 12:05:25,885 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 7b841029b07ea6e492c2c28c4a93a738, server=jenkins-hbase4.apache.org,35959,1689768323491}] 2023-07-19 12:05:26,039 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:26,040 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 12:05:26,043 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40548, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 12:05:26,049 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:26,050 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7b841029b07ea6e492c2c28c4a93a738, NAME => 'hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:26,051 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:26,051 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:26,051 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:26,051 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:26,053 INFO [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:26,056 DEBUG [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/info 2023-07-19 12:05:26,056 DEBUG [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/info 2023-07-19 12:05:26,056 INFO [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7b841029b07ea6e492c2c28c4a93a738 columnFamilyName info 2023-07-19 12:05:26,057 INFO [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] regionserver.HStore(310): Store=7b841029b07ea6e492c2c28c4a93a738/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:26,059 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:26,060 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:26,065 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:26,069 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 12:05:26,070 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7b841029b07ea6e492c2c28c4a93a738; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71328838, jitterRate=0.06288251280784607}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 12:05:26,070 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7b841029b07ea6e492c2c28c4a93a738: 2023-07-19 12:05:26,073 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738., pid=6, masterSystemTime=1689768326039 2023-07-19 12:05:26,079 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:26,079 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:26,081 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7b841029b07ea6e492c2c28c4a93a738, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:26,081 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689768326080"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689768326080"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689768326080"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689768326080"}]},"ts":"1689768326080"} 2023-07-19 12:05:26,089 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-19 12:05:26,090 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 7b841029b07ea6e492c2c28c4a93a738, server=jenkins-hbase4.apache.org,35959,1689768323491 in 201 msec 2023-07-19 12:05:26,094 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-19 12:05:26,094 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN in 369 msec 2023-07-19 12:05:26,096 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-19 12:05:26,096 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689768326096"}]},"ts":"1689768326096"} 2023-07-19 12:05:26,099 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-19 12:05:26,102 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-19 12:05:26,105 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 533 msec 2023-07-19 12:05:26,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-19 12:05:26,181 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-19 12:05:26,181 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:26,202 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:26,206 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40556, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:26,227 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-19 12:05:26,255 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-19 12:05:26,262 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 47 msec 2023-07-19 12:05:26,272 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-19 12:05:26,284 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-19 12:05:26,290 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-07-19 12:05:26,299 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-19 12:05:26,302 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-19 12:05:26,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.756sec 2023-07-19 12:05:26,306 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-19 12:05:26,308 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-19 12:05:26,308 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-19 12:05:26,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37041,1689768322202-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-19 12:05:26,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37041,1689768322202-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-19 12:05:26,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-19 12:05:26,320 DEBUG [Listener at localhost/34139] zookeeper.ReadOnlyZKClient(139): Connect 0x7ad49168 to 127.0.0.1:53047 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 12:05:26,330 DEBUG [Listener at localhost/34139] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19426020, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 12:05:26,348 DEBUG [hconnection-0x523b9f44-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:26,360 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48516, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:26,370 INFO [Listener at localhost/34139] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:26,381 DEBUG [Listener at localhost/34139] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-19 12:05:26,384 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51894, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-19 12:05:26,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-19 12:05:26,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-19 12:05:26,405 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-19 12:05:26,407 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-19 12:05:26,410 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp/data/default/t/78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:26,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-19 12:05:26,411 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp/data/default/t/78e0d689b75ff21c463fc267bf77dc89 empty. 2023-07-19 12:05:26,413 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp/data/default/t/78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:26,413 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-19 12:05:26,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 12:05:26,435 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-19 12:05:26,437 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 78e0d689b75ff21c463fc267bf77dc89, NAME => 't,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/.tmp 2023-07-19 12:05:26,457 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:26,458 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 78e0d689b75ff21c463fc267bf77dc89, disabling compactions & flushes 2023-07-19 12:05:26,458 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:26,458 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:26,458 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. after waiting 0 ms 2023-07-19 12:05:26,458 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:26,458 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:26,458 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 78e0d689b75ff21c463fc267bf77dc89: 2023-07-19 12:05:26,463 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-19 12:05:26,464 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689768326464"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689768326464"}]},"ts":"1689768326464"} 2023-07-19 12:05:26,467 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-19 12:05:26,469 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-19 12:05:26,469 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689768326469"}]},"ts":"1689768326469"} 2023-07-19 12:05:26,471 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-19 12:05:26,475 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 12:05:26,477 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 12:05:26,477 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 12:05:26,477 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 12:05:26,477 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN}] 2023-07-19 12:05:26,479 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN 2023-07-19 12:05:26,480 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35959,1689768323491; forceNewPlan=false, retain=false 2023-07-19 12:05:26,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 12:05:26,631 INFO [jenkins-hbase4:37041] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-19 12:05:26,632 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=78e0d689b75ff21c463fc267bf77dc89, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:26,632 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689768326632"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689768326632"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689768326632"}]},"ts":"1689768326632"} 2023-07-19 12:05:26,636 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 78e0d689b75ff21c463fc267bf77dc89, server=jenkins-hbase4.apache.org,35959,1689768323491}] 2023-07-19 12:05:26,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 12:05:26,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:26,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 78e0d689b75ff21c463fc267bf77dc89, NAME => 't,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:26,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:26,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:26,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:26,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:26,800 INFO [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:26,803 DEBUG [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/f 2023-07-19 12:05:26,803 DEBUG [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/f 2023-07-19 12:05:26,804 INFO [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78e0d689b75ff21c463fc267bf77dc89 columnFamilyName f 2023-07-19 12:05:26,804 INFO [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] regionserver.HStore(310): Store=78e0d689b75ff21c463fc267bf77dc89/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:26,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:26,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:26,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:26,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 12:05:26,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 78e0d689b75ff21c463fc267bf77dc89; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60106397, jitterRate=-0.10434488952159882}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 12:05:26,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 78e0d689b75ff21c463fc267bf77dc89: 2023-07-19 12:05:26,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89., pid=11, masterSystemTime=1689768326789 2023-07-19 12:05:26,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:26,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:26,821 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=78e0d689b75ff21c463fc267bf77dc89, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:26,822 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689768326821"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689768326821"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689768326821"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689768326821"}]},"ts":"1689768326821"} 2023-07-19 12:05:26,827 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-19 12:05:26,828 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 78e0d689b75ff21c463fc267bf77dc89, server=jenkins-hbase4.apache.org,35959,1689768323491 in 188 msec 2023-07-19 12:05:26,831 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-19 12:05:26,831 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN in 351 msec 2023-07-19 12:05:26,832 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-19 12:05:26,832 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689768326832"}]},"ts":"1689768326832"} 2023-07-19 12:05:26,834 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-19 12:05:26,837 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-19 12:05:26,840 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 439 msec 2023-07-19 12:05:27,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 12:05:27,038 INFO [Listener at localhost/34139] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-19 12:05:27,038 DEBUG [Listener at localhost/34139] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-19 12:05:27,040 INFO [Listener at localhost/34139] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-19 12:05:27,045 INFO [Listener at localhost/34139] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-19 12:05:27,045 INFO [Listener at localhost/34139] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-19 12:05:27,045 INFO [Listener at localhost/34139] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-19 12:05:27,049 DEBUG [Listener at localhost/34139] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:27,084 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40572, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:27,131 INFO [Listener at localhost/34139] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=587, MaxFileDescriptor=60000, SystemLoadAverage=177, ProcessCount=170, AvailableMemoryMB=6992 2023-07-19 12:05:27,132 INFO [Listener at localhost/34139] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-19 12:05:27,136 DEBUG [Listener at localhost/34139] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 12:05:27,137 DEBUG [Listener at localhost/34139] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:27,137 INFO [Listener at localhost/34139] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-19 12:05:27,137 INFO [Listener at localhost/34139] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-19 12:05:27,142 INFO [Listener at localhost/34139] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689768326096/Put/vlen=2/seqid=0} 2023-07-19 12:05:27,143 INFO [Listener at localhost/34139] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-19 12:05:27,143 INFO [Listener at localhost/34139] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738./info:regioninfo/1689768326080/Put/vlen=41/seqid=0, hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738./info:seqnumDuringOpen/1689768326080/Put/vlen=8/seqid=0, hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738./info:server/1689768326080/Put/vlen=31/seqid=0, hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738./info:serverstartcode/1689768326080/Put/vlen=8/seqid=0, hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738./info:sn/1689768325880/Put/vlen=45/seqid=0, hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738./info:state/1689768326080/Put/vlen=4/seqid=0} 2023-07-19 12:05:27,143 INFO [Listener at localhost/34139] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 7b841029b07ea6e492c2c28c4a93a738, NAME => 'hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:27,143 INFO [Listener at localhost/34139] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689768326832/Put/vlen=2/seqid=0} 2023-07-19 12:05:27,143 INFO [Listener at localhost/34139] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-19 12:05:27,143 INFO [Listener at localhost/34139] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89./info:regioninfo/1689768326821/Put/vlen=35/seqid=0, t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89./info:seqnumDuringOpen/1689768326821/Put/vlen=8/seqid=0, t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89./info:server/1689768326821/Put/vlen=31/seqid=0, t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89./info:serverstartcode/1689768326821/Put/vlen=8/seqid=0, t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89./info:sn/1689768326632/Put/vlen=45/seqid=0, t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89./info:state/1689768326821/Put/vlen=4/seqid=0} 2023-07-19 12:05:27,144 INFO [Listener at localhost/34139] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 78e0d689b75ff21c463fc267bf77dc89, NAME => 't,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:27,145 DEBUG [Listener at localhost/34139] zookeeper.ReadOnlyZKClient(139): Connect 0x5c83906f to 127.0.0.1:53047 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 12:05:27,151 DEBUG [Listener at localhost/34139] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58f96b33, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 12:05:27,152 INFO [Listener at localhost/34139] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-19 12:05:27,152 INFO [Listener at localhost/34139] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-19 12:05:27,155 DEBUG [hconnection-0x2f7b538a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:27,159 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48530, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:27,174 DEBUG [hconnection-0x2f7b538a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:27,177 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40584, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:27,179 INFO [Listener at localhost/34139] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-19 12:05:27,182 ERROR [Listener at localhost/34139] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,35959,1689768323491: die! ***** 2023-07-19 12:05:27,182 ERROR [Listener at localhost/34139] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-19 12:05:27,186 DEBUG [Listener at localhost/34139] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-19 12:05:27,189 DEBUG [Listener at localhost/34139] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-19 12:05:27,189 DEBUG [Listener at localhost/34139] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-19 12:05:27,190 DEBUG [Listener at localhost/34139] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-19 12:05:27,190 INFO [Listener at localhost/34139] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 980418560, "init": 513802240, "max": 2051014656, "used": 150484152 }, "NonHeapMemoryUsage": { "committed": 113008640, "init": 2555904, "max": -1, "used": 111006680 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-19 12:05:27,194 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37041] master.MasterRpcServices(612): jenkins-hbase4.apache.org,35959,1689768323491 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,35959,1689768323491: die! ***** 2023-07-19 12:05:27,203 INFO [Listener at localhost/34139] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,35959,1689768323491' ***** 2023-07-19 12:05:27,204 INFO [Listener at localhost/34139] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-19 12:05:27,204 INFO [RS:1;jenkins-hbase4:35959] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 12:05:27,204 INFO [RS:1;jenkins-hbase4:35959] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-19 12:05:27,204 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 12:05:27,204 INFO [RS:1;jenkins-hbase4:35959] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-19 12:05:27,204 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(3305): Received CLOSE for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:27,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35959] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 5438041200806330370 2023-07-19 12:05:27,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35959,1689768323491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 12:05:27,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35959] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:40584 deadline: 1689768387205, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35959,1689768323491 aborting 2023-07-19 12:05:27,206 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(3305): Received CLOSE for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:27,207 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:27,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 78e0d689b75ff21c463fc267bf77dc89, disabling compactions & flushes 2023-07-19 12:05:27,208 DEBUG [RS:1;jenkins-hbase4:35959] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1dd9a9b6 to 127.0.0.1:53047 2023-07-19 12:05:27,208 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:27,208 DEBUG [RS:1;jenkins-hbase4:35959] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:27,208 DEBUG [Listener at localhost/34139] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause 2023-07-19 12:05:27,208 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1474): Waiting on 2 regions to close org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause 2023-07-19 12:05:27,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) 2023-07-19 12:05:27,209 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1478): Online Regions={78e0d689b75ff21c463fc267bf77dc89=t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89., 7b841029b07ea6e492c2c28c4a93a738=hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.} at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-19 12:05:27,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. after waiting 0 ms Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35959,1689768323491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 12:05:27,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35959,1689768323491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-19 12:05:27,212 DEBUG [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1504): Waiting on 78e0d689b75ff21c463fc267bf77dc89, 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:27,212 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:27,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:27,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 78e0d689b75ff21c463fc267bf77dc89: 2023-07-19 12:05:27,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:27,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7b841029b07ea6e492c2c28c4a93a738, disabling compactions & flushes 2023-07-19 12:05:27,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:27,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:27,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35959,1689768323491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 12:05:27,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. after waiting 0 ms 2023-07-19 12:05:27,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35959] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:40584 deadline: 1689768387214, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35959,1689768323491 aborting 2023-07-19 12:05:27,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:27,215 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:27,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:27,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7b841029b07ea6e492c2c28c4a93a738: 2023-07-19 12:05:27,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:27,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35959,1689768323491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 12:05:27,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35959] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:40584 deadline: 1689768387327, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35959,1689768323491 aborting 2023-07-19 12:05:27,412 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,35959,1689768323491; all regions closed. 2023-07-19 12:05:27,424 DEBUG [RS:1;jenkins-hbase4:35959] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:27,425 INFO [RS:1;jenkins-hbase4:35959] regionserver.LeaseManager(133): Closed leases 2023-07-19 12:05:27,427 INFO [RS:1;jenkins-hbase4:35959] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-19 12:05:27,427 INFO [RS:1;jenkins-hbase4:35959] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 12:05:27,427 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 12:05:27,427 INFO [RS:1;jenkins-hbase4:35959] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 12:05:27,428 INFO [RS:1;jenkins-hbase4:35959] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 12:05:27,429 INFO [RS:1;jenkins-hbase4:35959] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35959 2023-07-19 12:05:27,443 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:27,443 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:27,443 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:27,443 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:27,443 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:27,444 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35959,1689768323491] 2023-07-19 12:05:27,444 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35959,1689768323491; numProcessing=1 2023-07-19 12:05:27,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:27,446 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,35959,1689768323491 znode expired, triggering replicatorRemoved event 2023-07-19 12:05:27,448 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35959,1689768323491 already deleted, retry=false 2023-07-19 12:05:27,448 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,35959,1689768323491 on jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:27,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:27,459 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,35959,1689768323491, splitWal=true, meta=false 2023-07-19 12:05:27,459 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,35959,1689768323491 (carryingMeta=false) jenkins-hbase4.apache.org,35959,1689768323491/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@5a277ba9[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-19 12:05:27,463 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35959,1689768323491, splitWal=true, meta=false 2023-07-19 12:05:27,465 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,35959,1689768323491 had 2 regions 2023-07-19 12:05:27,467 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35959,1689768323491, splitWal=true, meta=false, isMeta: false 2023-07-19 12:05:27,469 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting 2023-07-19 12:05:27,473 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,35959,1689768323491 WAL count=1, meta=false 2023-07-19 12:05:27,476 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088}] 2023-07-19 12:05:27,485 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:27,487 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088, worker=jenkins-hbase4.apache.org,39213,1689768323446}] 2023-07-19 12:05:27,534 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35959 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35959 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 12:05:27,535 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35959 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35959 2023-07-19 12:05:27,547 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 12:05:27,548 INFO [RS:1;jenkins-hbase4:35959] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,35959,1689768323491; zookeeper connection closed. 2023-07-19 12:05:27,548 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:35959-0x1017d78e43d0002, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 12:05:27,548 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5ffa3380] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5ffa3380 2023-07-19 12:05:27,654 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39213] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-19 12:05:27,677 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088, size=1.6 K (1665bytes) 2023-07-19 12:05:27,677 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088 2023-07-19 12:05:27,682 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088 after 4ms 2023-07-19 12:05:27,687 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088 took 12ms 2023-07-19 12:05:27,702 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 7b841029b07ea6e492c2c28c4a93a738: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-19 12:05:27,705 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 78e0d689b75ff21c463fc267bf77dc89: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-19 12:05:27,706 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088 so closing down 2023-07-19 12:05:27,706 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 12:05:27,713 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-0] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-19 12:05:27,717 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp 2023-07-19 12:05:27,717 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-0] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp 2023-07-19 12:05:27,719 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 12:05:27,735 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp (wrote 10 edits, skipped 0 edits in 1 ms) 2023-07-19 12:05:27,735 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp (wrote 2 edits, skipped 0 edits in 1 ms) 2023-07-19 12:05:27,737 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000013 2023-07-19 12:05:27,737 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000005 2023-07-19 12:05:27,738 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 50 ms; skipped=2; WAL=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088, size=1.6 K, length=1665, corrupted=false, cancelled=false 2023-07-19 12:05:27,738 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088, journal: Splitting hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088, size=1.6 K (1665bytes) at 1689768327677Finishing writing output for hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088 so closing down at 1689768327706 (+29 ms)Creating recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp at 1689768327717 (+11 ms)Creating recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp at 16897683277173 split writer threads finished at 1689768327719 (+2 ms)Closed recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp (wrote 10 edits, skipped 0 edits in 1 ms) at 1689768327735 (+16 ms)Closed recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp (wrote 2 edits, skipped 0 edits in 1 ms) at 1689768327735Rename recovered edits hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000013 at 1689768327737 (+2 ms)Rename recovered edits hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088.temp to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000005 at 1689768327738 (+1 ms)Processed 14 edits across 2 Regions in 50 ms; skipped=2; WAL=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088, size=1.6 K, length=1665, corrupted=false, cancelled=false at 1689768327738 2023-07-19 12:05:27,739 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088 2023-07-19 12:05:27,741 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-19 12:05:27,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] master.HMaster(3641): Remote procedure done, pid=14 2023-07-19 12:05:27,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] wal.WALSplitUtil(141): Moved hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting/jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088 to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs 2023-07-19 12:05:27,755 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-19 12:05:27,755 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088, worker=jenkins-hbase4.apache.org,39213,1689768323446 in 265 msec 2023-07-19 12:05:27,757 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:27,761 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-19 12:05:27,761 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C35959%2C1689768323491.1689768325088, worker=jenkins-hbase4.apache.org,39213,1689768323446 in 283 msec 2023-07-19 12:05:27,764 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting dir is empty, no logs to split. 2023-07-19 12:05:27,764 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,35959,1689768323491 WAL count=0, meta=false 2023-07-19 12:05:27,764 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,35959,1689768323491 WAL splitting is done? wals=0, meta=false 2023-07-19 12:05:27,767 INFO [PEWorker-5] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,35959,1689768323491 failed, ignore...File hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,35959,1689768323491-splitting does not exist. 2023-07-19 12:05:27,769 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN}] 2023-07-19 12:05:27,770 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN 2023-07-19 12:05:27,771 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN 2023-07-19 12:05:27,772 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 12:05:27,772 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 12:05:27,772 DEBUG [jenkins-hbase4:37041] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 12:05:27,773 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=78e0d689b75ff21c463fc267bf77dc89, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:27,773 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=7b841029b07ea6e492c2c28c4a93a738, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:27,774 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689768327773"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689768327773"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689768327773"}]},"ts":"1689768327773"} 2023-07-19 12:05:27,774 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689768327773"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689768327773"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689768327773"}]},"ts":"1689768327773"} 2023-07-19 12:05:27,778 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 78e0d689b75ff21c463fc267bf77dc89, server=jenkins-hbase4.apache.org,39213,1689768323446}] 2023-07-19 12:05:27,780 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=15, state=RUNNABLE; OpenRegionProcedure 7b841029b07ea6e492c2c28c4a93a738, server=jenkins-hbase4.apache.org,39213,1689768323446}] 2023-07-19 12:05:27,842 DEBUG [hconnection-0x2f7b538a-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35959 this server is in the failed servers list 2023-07-19 12:05:27,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:27,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 78e0d689b75ff21c463fc267bf77dc89, NAME => 't,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:27,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:27,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:27,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:27,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:27,949 INFO [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:27,952 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:27,952 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7b841029b07ea6e492c2c28c4a93a738, NAME => 'hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:27,952 DEBUG [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/f 2023-07-19 12:05:27,952 DEBUG [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/f 2023-07-19 12:05:27,952 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:27,952 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:27,952 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:27,953 INFO [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78e0d689b75ff21c463fc267bf77dc89 columnFamilyName f 2023-07-19 12:05:27,953 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:27,953 INFO [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] regionserver.HStore(310): Store=78e0d689b75ff21c463fc267bf77dc89/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:27,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:27,956 INFO [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:27,957 DEBUG [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/info 2023-07-19 12:05:27,957 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 12:05:27,957 DEBUG [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/info 2023-07-19 12:05:27,958 INFO [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7b841029b07ea6e492c2c28c4a93a738 columnFamilyName info 2023-07-19 12:05:27,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:27,959 INFO [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] regionserver.HStore(310): Store=7b841029b07ea6e492c2c28c4a93a738/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:27,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000013 2023-07-19 12:05:27,963 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:27,967 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:27,969 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000005 2023-07-19 12:05:27,972 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000013 2023-07-19 12:05:27,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 78e0d689b75ff21c463fc267bf77dc89 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-19 12:05:27,974 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000005 2023-07-19 12:05:27,974 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7b841029b07ea6e492c2c28c4a93a738 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-19 12:05:28,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/.tmp/f/2454a50952f34db59ac297a0e0302268 2023-07-19 12:05:28,044 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/.tmp/info/6b65f65c8e5d459ab3a8356266fe30d8 2023-07-19 12:05:28,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/.tmp/f/2454a50952f34db59ac297a0e0302268 as hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/f/2454a50952f34db59ac297a0e0302268 2023-07-19 12:05:28,090 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/.tmp/info/6b65f65c8e5d459ab3a8356266fe30d8 as hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/info/6b65f65c8e5d459ab3a8356266fe30d8 2023-07-19 12:05:28,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/f/2454a50952f34db59ac297a0e0302268, entries=10, sequenceid=13, filesize=4.8 K 2023-07-19 12:05:28,100 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/info/6b65f65c8e5d459ab3a8356266fe30d8, entries=2, sequenceid=5, filesize=4.8 K 2023-07-19 12:05:28,101 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 7b841029b07ea6e492c2c28c4a93a738 in 127ms, sequenceid=5, compaction requested=false; wal=null 2023-07-19 12:05:28,101 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 78e0d689b75ff21c463fc267bf77dc89 in 129ms, sequenceid=13, compaction requested=false; wal=null 2023-07-19 12:05:28,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-19 12:05:28,103 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-19 12:05:28,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/0000000000000000013 2023-07-19 12:05:28,106 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/0000000000000000005 2023-07-19 12:05:28,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:28,112 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:28,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-19 12:05:28,117 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-19 12:05:28,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 78e0d689b75ff21c463fc267bf77dc89; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65654097, jitterRate=-0.021677717566490173}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 12:05:28,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 78e0d689b75ff21c463fc267bf77dc89: 2023-07-19 12:05:28,118 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7b841029b07ea6e492c2c28c4a93a738; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59625007, jitterRate=-0.11151815950870514}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 12:05:28,118 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7b841029b07ea6e492c2c28c4a93a738: 2023-07-19 12:05:28,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89., pid=17, masterSystemTime=1689768327940 2023-07-19 12:05:28,119 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738., pid=18, masterSystemTime=1689768327940 2023-07-19 12:05:28,122 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:28,122 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:28,123 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=7b841029b07ea6e492c2c28c4a93a738, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:28,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:28,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:28,123 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689768328123"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689768328123"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689768328123"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689768328123"}]},"ts":"1689768328123"} 2023-07-19 12:05:28,130 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=78e0d689b75ff21c463fc267bf77dc89, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:28,130 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689768328130"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689768328130"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689768328130"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689768328130"}]},"ts":"1689768328130"} 2023-07-19 12:05:28,132 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=15 2023-07-19 12:05:28,134 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=15, state=SUCCESS; OpenRegionProcedure 7b841029b07ea6e492c2c28c4a93a738, server=jenkins-hbase4.apache.org,39213,1689768323446 in 346 msec 2023-07-19 12:05:28,136 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN in 365 msec 2023-07-19 12:05:28,138 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-07-19 12:05:28,138 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 78e0d689b75ff21c463fc267bf77dc89, server=jenkins-hbase4.apache.org,39213,1689768323446 in 356 msec 2023-07-19 12:05:28,141 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=12 2023-07-19 12:05:28,141 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,35959,1689768323491 after splitting done 2023-07-19 12:05:28,141 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN in 369 msec 2023-07-19 12:05:28,141 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,35959,1689768323491 from processing; numProcessing=0 2023-07-19 12:05:28,144 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,35959,1689768323491, splitWal=true, meta=false in 691 msec 2023-07-19 12:05:28,364 DEBUG [Listener at localhost/34139] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c83906f to 127.0.0.1:53047 2023-07-19 12:05:28,364 DEBUG [Listener at localhost/34139] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:28,364 INFO [Listener at localhost/34139] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-19 12:05:28,379 INFO [Listener at localhost/34139] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=273 (was 294), OpenFileDescriptor=575 (was 587), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=177 (was 177), ProcessCount=170 (was 170), AvailableMemoryMB=6913 (was 6992) 2023-07-19 12:05:28,401 INFO [Listener at localhost/34139] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=273, OpenFileDescriptor=575, MaxFileDescriptor=60000, SystemLoadAverage=177, ProcessCount=170, AvailableMemoryMB=6895 2023-07-19 12:05:28,402 INFO [Listener at localhost/34139] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:35959 2023-07-19 12:05:28,415 INFO [Listener at localhost/34139] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 12:05:28,415 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:28,415 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:28,415 INFO [Listener at localhost/34139] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 12:05:28,416 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 12:05:28,416 INFO [Listener at localhost/34139] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 12:05:28,416 INFO [Listener at localhost/34139] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 12:05:28,418 INFO [Listener at localhost/34139] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43697 2023-07-19 12:05:28,419 INFO [Listener at localhost/34139] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 12:05:28,420 DEBUG [Listener at localhost/34139] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 12:05:28,421 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:28,422 INFO [Listener at localhost/34139] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 12:05:28,423 INFO [Listener at localhost/34139] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43697 connecting to ZooKeeper ensemble=127.0.0.1:53047 2023-07-19 12:05:28,427 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:436970x0, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 12:05:28,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43697-0x1017d78e43d0008 connected 2023-07-19 12:05:28,428 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(162): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 12:05:28,429 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(162): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-19 12:05:28,430 DEBUG [Listener at localhost/34139] zookeeper.ZKUtil(164): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 12:05:28,430 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43697 2023-07-19 12:05:28,430 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43697 2023-07-19 12:05:28,431 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43697 2023-07-19 12:05:28,431 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43697 2023-07-19 12:05:28,432 DEBUG [Listener at localhost/34139] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43697 2023-07-19 12:05:28,434 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(951): ClusterId : fc90d035-6952-4500-aaae-cb5f010aeb03 2023-07-19 12:05:28,435 DEBUG [RS:2;jenkins-hbase4:43697] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 12:05:28,437 DEBUG [RS:2;jenkins-hbase4:43697] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 12:05:28,437 DEBUG [RS:2;jenkins-hbase4:43697] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 12:05:28,439 DEBUG [RS:2;jenkins-hbase4:43697] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 12:05:28,440 DEBUG [RS:2;jenkins-hbase4:43697] zookeeper.ReadOnlyZKClient(139): Connect 0x0940959a to 127.0.0.1:53047 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 12:05:28,445 DEBUG [RS:2;jenkins-hbase4:43697] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7877ac2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 12:05:28,446 DEBUG [RS:2;jenkins-hbase4:43697] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b8feea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 12:05:28,454 DEBUG [RS:2;jenkins-hbase4:43697] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:43697 2023-07-19 12:05:28,454 INFO [RS:2;jenkins-hbase4:43697] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 12:05:28,454 INFO [RS:2;jenkins-hbase4:43697] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 12:05:28,454 DEBUG [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 12:05:28,455 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,37041,1689768322202 with isa=jenkins-hbase4.apache.org/172.31.14.131:43697, startcode=1689768328415 2023-07-19 12:05:28,455 DEBUG [RS:2;jenkins-hbase4:43697] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 12:05:28,458 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52593, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 12:05:28,459 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37041] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:28,459 DEBUG [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d 2023-07-19 12:05:28,459 DEBUG [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:41359 2023-07-19 12:05:28,459 DEBUG [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 12:05:28,461 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:28,461 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:28,461 DEBUG [RS:2;jenkins-hbase4:43697] zookeeper.ZKUtil(162): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:28,461 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43697,1689768328415] 2023-07-19 12:05:28,461 WARN [RS:2;jenkins-hbase4:43697] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 12:05:28,462 INFO [RS:2;jenkins-hbase4:43697] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 12:05:28,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:28,462 DEBUG [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1948): logDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:28,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:28,467 DEBUG [RS:2;jenkins-hbase4:43697] zookeeper.ZKUtil(162): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:28,468 DEBUG [RS:2;jenkins-hbase4:43697] zookeeper.ZKUtil(162): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:28,469 DEBUG [RS:2;jenkins-hbase4:43697] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 12:05:28,470 INFO [RS:2;jenkins-hbase4:43697] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 12:05:28,471 INFO [RS:2;jenkins-hbase4:43697] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 12:05:28,472 INFO [RS:2;jenkins-hbase4:43697] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 12:05:28,472 INFO [RS:2;jenkins-hbase4:43697] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:28,472 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 12:05:28,474 INFO [RS:2;jenkins-hbase4:43697] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:28,474 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:28,474 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:28,474 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:28,474 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:28,474 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:28,474 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 12:05:28,474 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:28,475 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:28,475 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:28,475 DEBUG [RS:2;jenkins-hbase4:43697] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 12:05:28,483 INFO [RS:2;jenkins-hbase4:43697] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:28,483 INFO [RS:2;jenkins-hbase4:43697] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:28,483 INFO [RS:2;jenkins-hbase4:43697] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:28,494 INFO [RS:2;jenkins-hbase4:43697] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 12:05:28,495 INFO [RS:2;jenkins-hbase4:43697] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43697,1689768328415-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 12:05:28,507 INFO [RS:2;jenkins-hbase4:43697] regionserver.Replication(203): jenkins-hbase4.apache.org,43697,1689768328415 started 2023-07-19 12:05:28,507 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,43697,1689768328415, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43697, sessionid=0x1017d78e43d0008 2023-07-19 12:05:28,507 INFO [Listener at localhost/34139] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:43697,5,FailOnTimeoutGroup] 2023-07-19 12:05:28,507 DEBUG [RS:2;jenkins-hbase4:43697] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 12:05:28,507 DEBUG [RS:2;jenkins-hbase4:43697] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:28,507 DEBUG [RS:2;jenkins-hbase4:43697] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43697,1689768328415' 2023-07-19 12:05:28,507 DEBUG [RS:2;jenkins-hbase4:43697] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 12:05:28,507 INFO [Listener at localhost/34139] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:28,507 INFO [Listener at localhost/34139] client.TestScannerTimeout(177): START ************ test3686b 2023-07-19 12:05:28,508 DEBUG [RS:2;jenkins-hbase4:43697] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 12:05:28,508 DEBUG [RS:2;jenkins-hbase4:43697] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 12:05:28,508 DEBUG [RS:2;jenkins-hbase4:43697] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 12:05:28,508 DEBUG [RS:2;jenkins-hbase4:43697] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:28,508 DEBUG [RS:2;jenkins-hbase4:43697] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43697,1689768328415' 2023-07-19 12:05:28,508 DEBUG [RS:2;jenkins-hbase4:43697] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 12:05:28,509 DEBUG [RS:2;jenkins-hbase4:43697] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 12:05:28,510 DEBUG [RS:2;jenkins-hbase4:43697] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 12:05:28,510 INFO [RS:2;jenkins-hbase4:43697] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 12:05:28,510 INFO [RS:2;jenkins-hbase4:43697] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 12:05:28,510 DEBUG [Listener at localhost/34139] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 12:05:28,511 DEBUG [Listener at localhost/34139] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:28,511 DEBUG [Listener at localhost/34139] zookeeper.ReadOnlyZKClient(139): Connect 0x60505a67 to 127.0.0.1:53047 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 12:05:28,517 DEBUG [Listener at localhost/34139] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55d47cdc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 12:05:28,520 DEBUG [hconnection-0x10ba4d60-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:28,522 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48538, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:28,613 INFO [RS:2;jenkins-hbase4:43697] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43697%2C1689768328415, suffix=, logDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,43697,1689768328415, archiveDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs, maxLogs=32 2023-07-19 12:05:28,633 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK] 2023-07-19 12:05:28,633 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK] 2023-07-19 12:05:28,638 INFO [RS:2;jenkins-hbase4:43697] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,43697,1689768328415/jenkins-hbase4.apache.org%2C43697%2C1689768328415.1689768328614 2023-07-19 12:05:28,638 DEBUG [RS:2;jenkins-hbase4:43697] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK], DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK]] 2023-07-19 12:05:30,880 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-19 12:05:30,941 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-19 12:05:32,372 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 12:05:32,373 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-19 12:05:43,527 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease -5555723433565749230 expired clientIPAndPort=172.31.14.131:48538, userName=jenkins, regionInfo=t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:45,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39213] regionserver.RSRpcServices(3094): Client tried to access missing scanner -5555723433565749230 2023-07-19 12:05:45,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39213] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:48538 deadline: 1689768405529, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-5555723433565749230'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-19 12:05:45,531 DEBUG [Listener at localhost/34139] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-5555723433565749230'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-5555723433565749230'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-5555723433565749230'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-19 12:05:45,534 DEBUG [Listener at localhost/34139] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x60505a67 to 127.0.0.1:53047 2023-07-19 12:05:45,535 DEBUG [Listener at localhost/34139] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:45,535 INFO [Listener at localhost/34139] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-19 12:05:45,551 INFO [Listener at localhost/34139] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=297 (was 273) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-698320902_17 at /127.0.0.1:55454 [Receiving block BP-1088350189-172.31.14.131-1689768318955:blk_1073741843_1019] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53047@0x0940959a sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/550573279.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53047@0x0940959a-SendThread(127.0.0.1:53047) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:41359 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:43697-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53047@0x60505a67-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d-prefix:jenkins-hbase4.apache.org,43697,1689768328415 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-1088350189-172.31.14.131-1689768318955:blk_1073741843_1019, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53047@0x60505a67-SendThread(127.0.0.1:53047) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: jenkins-hbase4:43697Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53047@0x60505a67 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/550573279.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43697 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-1088350189-172.31.14.131-1689768318955:blk_1073741843_1019, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53047@0x0940959a-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS:2;jenkins-hbase4:43697 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-698320902_17 at /127.0.0.1:47812 [Receiving block BP-1088350189-172.31.14.131-1689768318955:blk_1073741843_1019] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=572 (was 575), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=172 (was 177), ProcessCount=170 (was 170), AvailableMemoryMB=6686 (was 6895) 2023-07-19 12:05:45,569 INFO [Listener at localhost/34139] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=297, OpenFileDescriptor=572, MaxFileDescriptor=60000, SystemLoadAverage=172, ProcessCount=170, AvailableMemoryMB=6681 2023-07-19 12:05:45,570 INFO [Listener at localhost/34139] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:35959 2023-07-19 12:05:45,570 INFO [Listener at localhost/34139] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,35959,1689768323491 2023-07-19 12:05:45,570 INFO [Listener at localhost/34139] client.TestScannerTimeout(104): START************ test2772 2023-07-19 12:05:45,573 DEBUG [Listener at localhost/34139] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 12:05:45,574 DEBUG [Listener at localhost/34139] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:45,574 DEBUG [Listener at localhost/34139] zookeeper.ReadOnlyZKClient(139): Connect 0x776125a4 to 127.0.0.1:53047 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 12:05:45,579 DEBUG [Listener at localhost/34139] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56417cbe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 12:05:45,580 ERROR [Listener at localhost/34139] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,39213,1689768323446: die! ***** 2023-07-19 12:05:45,580 ERROR [Listener at localhost/34139] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-19 12:05:45,580 DEBUG [Listener at localhost/34139] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-19 12:05:45,581 DEBUG [Listener at localhost/34139] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-19 12:05:45,586 DEBUG [Listener at localhost/34139] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-19 12:05:45,588 DEBUG [Listener at localhost/34139] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-19 12:05:45,610 INFO [Listener at localhost/34139] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 980418560, "init": 513802240, "max": 2051014656, "used": 276497888 }, "NonHeapMemoryUsage": { "committed": 122904576, "init": 2555904, "max": -1, "used": 120580944 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13024, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 63, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 63, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 63, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 63, "ResponseSize_min": 197, "ResponseSize_max": 679, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 199, "ResponseSize_median": 202, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 678, "ResponseSize_95th_percentile": 678, "ResponseSize_98th_percentile": 678, "ResponseSize_99th_percentile": 678, "ResponseSize_99.9th_percentile": 678, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 63, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16705 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:53047", "tag.serverName": "jenkins-hbase4.apache.org,39213,1689768323446", "tag.clusterId": "fc90d035-6952-4500-aaae-cb5f010aeb03", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 2, "maxStoreFileCount": 1, "memStoreSize": 4186, "storeFileSize": 9852, "maxStoreFileAge": 16805, "minStoreFileAge": 16804, "avgStoreFileAge": 16804, "numReferenceFiles": 0, "regionServerStartTime": 1689768323446, "averageRegionSize": 4679, "storeFileIndexSize": 640, "staticIndexSize": 68, "staticBloomSize": 4, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1251, "localBytesRead": 1251, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 3314, "ByteBuffAllocatorPoolAllocationBytes": 1530880, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 41, "totalRowActionRequestCount": 51, "readRequestCount": 37, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 6, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 21, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 12, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 348, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 20, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 9852, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 20, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 348, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.603496271273436, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.2535353349778684, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-19 12:05:45,611 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37041] master.MasterRpcServices(612): jenkins-hbase4.apache.org,39213,1689768323446 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,39213,1689768323446: die! ***** 2023-07-19 12:05:45,614 INFO [Listener at localhost/34139] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39213,1689768323446' ***** 2023-07-19 12:05:45,615 INFO [Listener at localhost/34139] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-19 12:05:45,615 INFO [RS:0;jenkins-hbase4:39213] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 12:05:45,615 INFO [RS:0;jenkins-hbase4:39213] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-19 12:05:45,615 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 12:05:45,615 INFO [RS:0;jenkins-hbase4:39213] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-19 12:05:45,616 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(3305): Received CLOSE for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:45,618 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(3305): Received CLOSE for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:45,619 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:45,619 DEBUG [RS:0;jenkins-hbase4:39213] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5bf435c3 to 127.0.0.1:53047 2023-07-19 12:05:45,619 DEBUG [RS:0;jenkins-hbase4:39213] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:45,619 INFO [RS:0;jenkins-hbase4:39213] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 12:05:45,619 INFO [RS:0;jenkins-hbase4:39213] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 12:05:45,619 INFO [RS:0;jenkins-hbase4:39213] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 12:05:45,619 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-19 12:05:45,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 78e0d689b75ff21c463fc267bf77dc89, disabling compactions & flushes 2023-07-19 12:05:45,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:45,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:45,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. after waiting 0 ms 2023-07-19 12:05:45,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:45,627 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-19 12:05:45,631 DEBUG [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1478): Online Regions={78e0d689b75ff21c463fc267bf77dc89=t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89., 1588230740=hbase:meta,,1.1588230740, 7b841029b07ea6e492c2c28c4a93a738=hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.} 2023-07-19 12:05:45,631 DEBUG [hconnection-0x7cfe3418-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:45,631 DEBUG [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1504): Waiting on 1588230740, 78e0d689b75ff21c463fc267bf77dc89, 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:45,635 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 12:05:45,635 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 12:05:45,635 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 12:05:45,635 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 12:05:45,635 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 12:05:45,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:45,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 78e0d689b75ff21c463fc267bf77dc89: 2023-07-19 12:05:45,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:45,636 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 4186 in region hbase:meta,,1.1588230740 2023-07-19 12:05:45,636 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38320, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:45,637 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 12:05:45,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7b841029b07ea6e492c2c28c4a93a738, disabling compactions & flushes 2023-07-19 12:05:45,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:45,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:45,637 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39213] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39213,1689768323446 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 12:05:45,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. after waiting 0 ms 2023-07-19 12:05:45,638 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39213] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:38320 deadline: 1689768405637, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39213,1689768323446 aborting 2023-07-19 12:05:45,638 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 12:05:45,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:45,638 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 12:05:45,639 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-19 12:05:45,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:45,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7b841029b07ea6e492c2c28c4a93a738: 2023-07-19 12:05:45,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:45,742 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39213] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39213,1689768323446 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 12:05:45,742 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39213] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:38320 deadline: 1689768405742, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39213,1689768323446 aborting 2023-07-19 12:05:45,831 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,39213,1689768323446; all regions closed. 2023-07-19 12:05:45,844 DEBUG [RS:0;jenkins-hbase4:39213] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:45,844 INFO [RS:0;jenkins-hbase4:39213] regionserver.LeaseManager(133): Closed leases 2023-07-19 12:05:45,845 INFO [RS:0;jenkins-hbase4:39213] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-19 12:05:45,845 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 12:05:45,846 INFO [RS:0;jenkins-hbase4:39213] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39213 2023-07-19 12:05:45,850 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:45,850 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 2023-07-19 12:05:45,850 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:45,850 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:45,850 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:45,851 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39213,1689768323446] 2023-07-19 12:05:45,851 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39213,1689768323446; numProcessing=1 2023-07-19 12:05:45,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:45,852 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,39213,1689768323446 znode expired, triggering replicatorRemoved event 2023-07-19 12:05:45,853 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39213,1689768323446 already deleted, retry=false 2023-07-19 12:05:45,853 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,39213,1689768323446 on jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:45,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:45,855 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,39213,1689768323446, splitWal=true, meta=true 2023-07-19 12:05:45,855 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=19 for jenkins-hbase4.apache.org,39213,1689768323446 (carryingMeta=true) jenkins-hbase4.apache.org,39213,1689768323446/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@338b386d[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-19 12:05:45,857 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=19, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39213,1689768323446, splitWal=true, meta=true 2023-07-19 12:05:45,858 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39213,1689768323446, splitWal=true, meta=true, isMeta: true 2023-07-19 12:05:45,862 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting 2023-07-19 12:05:45,864 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,39213,1689768323446 WAL count=1, meta=true 2023-07-19 12:05:45,864 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta}] 2023-07-19 12:05:45,867 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:45,868 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta, worker=jenkins-hbase4.apache.org,43697,1689768328415}] 2023-07-19 12:05:45,946 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39213 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39213 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 12:05:45,947 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39213 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39213 2023-07-19 12:05:45,957 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 12:05:45,957 INFO [RS:0;jenkins-hbase4:39213] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39213,1689768323446; zookeeper connection closed. 2023-07-19 12:05:45,957 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:39213-0x1017d78e43d0001, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 12:05:45,957 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7f8e25e5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7f8e25e5 2023-07-19 12:05:46,020 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:46,021 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 12:05:46,022 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 12:05:46,023 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43697] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=21 2023-07-19 12:05:46,039 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta, size=5.1 K (5185bytes) 2023-07-19 12:05:46,039 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta 2023-07-19 12:05:46,040 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta after 1ms 2023-07-19 12:05:46,045 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta took 7ms 2023-07-19 12:05:46,049 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-19 12:05:46,051 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta so closing down 2023-07-19 12:05:46,051 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 12:05:46,056 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta.temp 2023-07-19 12:05:46,057 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 12:05:46,070 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta.temp (wrote 14 edits, skipped 0 edits in 0 ms) 2023-07-19 12:05:46,072 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta.temp to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-19 12:05:46,072 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 15 edits across 1 Regions in 27 ms; skipped=1; WAL=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta, size=5.1 K, length=5185, corrupted=false, cancelled=false 2023-07-19 12:05:46,073 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta, journal: Splitting hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta, size=5.1 K (5185bytes) at 1689768346039Finishing writing output for hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta so closing down at 1689768346051 (+12 ms)Creating recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta.temp at 1689768346056 (+5 ms)3 split writer threads finished at 1689768346057 (+1 ms)Closed recovered edits writer path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta.temp (wrote 14 edits, skipped 0 edits in 0 ms) at 1689768346070 (+13 ms)Rename recovered edits hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta.temp to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 at 1689768346072 (+2 ms)Processed 15 edits across 1 Regions in 27 ms; skipped=1; WAL=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta, size=5.1 K, length=5185, corrupted=false, cancelled=false at 1689768346073 (+1 ms) 2023-07-19 12:05:46,073 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta 2023-07-19 12:05:46,073 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=21 2023-07-19 12:05:46,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] master.HMaster(3641): Remote procedure done, pid=21 2023-07-19 12:05:46,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] wal.WALSplitUtil(141): Moved hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs 2023-07-19 12:05:46,080 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-19 12:05:46,080 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta, worker=jenkins-hbase4.apache.org,43697,1689768328415 in 209 msec 2023-07-19 12:05:46,082 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:46,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-19 12:05:46,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C39213%2C1689768323446.meta.1689768325262.meta, worker=jenkins-hbase4.apache.org,43697,1689768328415 in 218 msec 2023-07-19 12:05:46,087 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting dir is empty, no logs to split. 2023-07-19 12:05:46,087 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,39213,1689768323446 WAL count=0, meta=true 2023-07-19 12:05:46,087 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39213,1689768323446 WAL splitting is done? wals=0, meta=true 2023-07-19 12:05:46,088 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-19 12:05:46,089 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-19 12:05:46,090 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 12:05:46,104 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 12:05:46,240 DEBUG [jenkins-hbase4:37041] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 12:05:46,241 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43697,1689768328415, state=OPENING 2023-07-19 12:05:46,242 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 12:05:46,243 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 12:05:46,243 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43697,1689768328415}] 2023-07-19 12:05:46,249 WARN [ReadOnlyZKClient-127.0.0.1:53047@0x776125a4] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-19 12:05:46,249 DEBUG [hconnection-0x7cfe3418-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:46,252 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:46,253 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43697] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:38772 deadline: 1689768406252, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:46,402 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-19 12:05:46,402 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 12:05:46,405 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43697%2C1689768328415.meta, suffix=.meta, logDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,43697,1689768328415, archiveDir=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs, maxLogs=32 2023-07-19 12:05:46,428 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK] 2023-07-19 12:05:46,428 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK] 2023-07-19 12:05:46,431 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,43697,1689768328415/jenkins-hbase4.apache.org%2C43697%2C1689768328415.meta.1689768346406.meta 2023-07-19 12:05:46,433 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42179,DS-d3550991-9b4a-4299-ae21-4c1c62fe2160,DISK], DatanodeInfoWithStorage[127.0.0.1:45357,DS-eade95c3-5bc2-422c-82de-f340614da657,DISK]] 2023-07-19 12:05:46,433 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:46,434 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-19 12:05:46,434 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-19 12:05:46,435 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-19 12:05:46,435 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-19 12:05:46,435 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:46,435 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-19 12:05:46,435 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-19 12:05:46,439 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 12:05:46,442 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/info 2023-07-19 12:05:46,442 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/info 2023-07-19 12:05:46,442 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 12:05:46,443 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:46,443 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 12:05:46,445 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/rep_barrier 2023-07-19 12:05:46,445 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/rep_barrier 2023-07-19 12:05:46,445 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 12:05:46,446 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:46,446 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 12:05:46,447 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/table 2023-07-19 12:05:46,447 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/table 2023-07-19 12:05:46,448 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 12:05:46,449 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:46,450 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740 2023-07-19 12:05:46,453 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740 2023-07-19 12:05:46,454 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-19 12:05:46,463 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 40, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=17, path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-19 12:05:46,463 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=4.09 KB heapSize=7.79 KB 2023-07-19 12:05:46,490 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.93 KB at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/.tmp/info/8ab15122f66c4d1f894d06a4eb8de776 2023-07-19 12:05:46,512 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/.tmp/table/a9d27adccba14edf87da44ac5fdaa01c 2023-07-19 12:05:46,520 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/.tmp/info/8ab15122f66c4d1f894d06a4eb8de776 as hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/info/8ab15122f66c4d1f894d06a4eb8de776 2023-07-19 12:05:46,529 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/info/8ab15122f66c4d1f894d06a4eb8de776, entries=36, sequenceid=17, filesize=8.6 K 2023-07-19 12:05:46,530 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/.tmp/table/a9d27adccba14edf87da44ac5fdaa01c as hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/table/a9d27adccba14edf87da44ac5fdaa01c 2023-07-19 12:05:46,538 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/table/a9d27adccba14edf87da44ac5fdaa01c, entries=4, sequenceid=17, filesize=4.7 K 2023-07-19 12:05:46,538 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~4.09 KB/4186, heapSize ~7.51 KB/7688, currentSize=0 B/0 for 1588230740 in 75ms, sequenceid=17, compaction requested=false; wal=null 2023-07-19 12:05:46,540 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-19 12:05:46,542 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 12:05:46,544 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 12:05:46,547 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=1 2023-07-19 12:05:46,548 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=18; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61336677, jitterRate=-0.08601228892803192}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 12:05:46,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 12:05:46,549 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=23, masterSystemTime=1689768346396 2023-07-19 12:05:46,552 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-19 12:05:46,552 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-19 12:05:46,553 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43697,1689768328415, state=OPEN 2023-07-19 12:05:46,554 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 12:05:46,555 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 12:05:46,557 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-19 12:05:46,557 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43697,1689768328415 in 312 msec 2023-07-19 12:05:46,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-07-19 12:05:46,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 469 msec 2023-07-19 12:05:46,560 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,39213,1689768323446 had 3 regions 2023-07-19 12:05:46,561 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39213,1689768323446, splitWal=true, meta=true, isMeta: false 2023-07-19 12:05:46,563 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39213,1689768323446 WAL count=1, meta=false 2023-07-19 12:05:46,563 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=19, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034}] 2023-07-19 12:05:46,565 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:46,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034, worker=jenkins-hbase4.apache.org,43697,1689768328415}] 2023-07-19 12:05:46,719 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43697] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-19 12:05:46,732 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034, size=721 (721bytes) 2023-07-19 12:05:46,732 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034 2023-07-19 12:05:46,733 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034 after 1ms 2023-07-19 12:05:46,736 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034 took 4ms 2023-07-19 12:05:46,747 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 78e0d689b75ff21c463fc267bf77dc89: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-19 12:05:46,749 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 7b841029b07ea6e492c2c28c4a93a738: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-19 12:05:46,749 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034 so closing down 2023-07-19 12:05:46,749 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 12:05:46,750 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 12:05:46,751 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 13 ms; skipped=2; WAL=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034, size=721, length=721, corrupted=false, cancelled=false 2023-07-19 12:05:46,751 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034, journal: Splitting hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034, size=721 (721bytes) at 1689768346732Finishing writing output for hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034 so closing down at 1689768346749 (+17 ms)3 split writer threads finished at 1689768346750 (+1 ms)Processed 2 edits across 0 Regions in 13 ms; skipped=2; WAL=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034, size=721, length=721, corrupted=false, cancelled=false at 1689768346751 (+1 ms) 2023-07-19 12:05:46,751 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034 2023-07-19 12:05:46,751 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-19 12:05:46,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] master.HMaster(3641): Remote procedure done, pid=25 2023-07-19 12:05:46,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37041] wal.WALSplitUtil(141): Moved hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting/jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034 to hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs 2023-07-19 12:05:46,758 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-19 12:05:46,759 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034, worker=jenkins-hbase4.apache.org,43697,1689768328415 in 190 msec 2023-07-19 12:05:46,761 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:46,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=19 2023-07-19 12:05:46,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=19, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C39213%2C1689768323446.1689768325034, worker=jenkins-hbase4.apache.org,43697,1689768328415 in 198 msec 2023-07-19 12:05:46,771 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/WALs/jenkins-hbase4.apache.org,39213,1689768323446-splitting dir is empty, no logs to split. 2023-07-19 12:05:46,771 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39213,1689768323446 WAL count=0, meta=false 2023-07-19 12:05:46,771 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39213,1689768323446 WAL splitting is done? wals=0, meta=false 2023-07-19 12:05:46,775 DEBUG [hconnection-0x7cfe3418-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39213 this server is in the failed servers list 2023-07-19 12:05:46,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN}, {pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN}] 2023-07-19 12:05:46,777 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN 2023-07-19 12:05:46,777 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN 2023-07-19 12:05:46,778 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 12:05:46,778 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 12:05:46,778 DEBUG [jenkins-hbase4:37041] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 12:05:46,780 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=7b841029b07ea6e492c2c28c4a93a738, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:46,780 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689768346780"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689768346780"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689768346780"}]},"ts":"1689768346780"} 2023-07-19 12:05:46,781 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=78e0d689b75ff21c463fc267bf77dc89, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:46,781 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689768346781"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689768346781"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689768346781"}]},"ts":"1689768346781"} 2023-07-19 12:05:46,781 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39213 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39213 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 12:05:46,783 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39213 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39213 2023-07-19 12:05:46,784 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39213 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39213 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 12:05:46,784 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39213 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39213 2023-07-19 12:05:46,885 DEBUG [PEWorker-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 12:05:46,887 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 12:05:46,887 DEBUG [hconnection-0x7cfe3418-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39213 this server is in the failed servers list 2023-07-19 12:05:46,890 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=26, state=RUNNABLE; OpenRegionProcedure 7b841029b07ea6e492c2c28c4a93a738, server=jenkins-hbase4.apache.org,43697,1689768328415}] 2023-07-19 12:05:46,891 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=27, state=RUNNABLE; OpenRegionProcedure 78e0d689b75ff21c463fc267bf77dc89, server=jenkins-hbase4.apache.org,43697,1689768328415}] 2023-07-19 12:05:47,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:47,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 78e0d689b75ff21c463fc267bf77dc89, NAME => 't,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:47,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:47,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:47,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:47,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:47,052 INFO [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:47,053 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:47,053 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7b841029b07ea6e492c2c28c4a93a738, NAME => 'hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.', STARTKEY => '', ENDKEY => ''} 2023-07-19 12:05:47,054 DEBUG [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/f 2023-07-19 12:05:47,054 DEBUG [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/f 2023-07-19 12:05:47,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:47,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 12:05:47,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:47,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:47,054 INFO [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 78e0d689b75ff21c463fc267bf77dc89 columnFamilyName f 2023-07-19 12:05:47,056 INFO [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:47,057 DEBUG [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/info 2023-07-19 12:05:47,057 DEBUG [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/info 2023-07-19 12:05:47,058 INFO [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7b841029b07ea6e492c2c28c4a93a738 columnFamilyName info 2023-07-19 12:05:47,064 DEBUG [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] regionserver.HStore(539): loaded hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/f/2454a50952f34db59ac297a0e0302268 2023-07-19 12:05:47,068 INFO [StoreOpener-78e0d689b75ff21c463fc267bf77dc89-1] regionserver.HStore(310): Store=78e0d689b75ff21c463fc267bf77dc89/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:47,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:47,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:47,071 DEBUG [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] regionserver.HStore(539): loaded hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/info/6b65f65c8e5d459ab3a8356266fe30d8 2023-07-19 12:05:47,071 INFO [StoreOpener-7b841029b07ea6e492c2c28c4a93a738-1] regionserver.HStore(310): Store=7b841029b07ea6e492c2c28c4a93a738/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 12:05:47,072 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:47,074 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:47,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:47,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-19 12:05:47,078 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:47,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 78e0d689b75ff21c463fc267bf77dc89; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63080325, jitterRate=-0.06002990901470184}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 12:05:47,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 78e0d689b75ff21c463fc267bf77dc89: 2023-07-19 12:05:47,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89., pid=29, masterSystemTime=1689768347045 2023-07-19 12:05:47,081 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-19 12:05:47,083 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7b841029b07ea6e492c2c28c4a93a738; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73357581, jitterRate=0.09311313927173615}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 12:05:47,083 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:47,083 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7b841029b07ea6e492c2c28c4a93a738: 2023-07-19 12:05:47,083 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:47,084 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738., pid=28, masterSystemTime=1689768347045 2023-07-19 12:05:47,084 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=78e0d689b75ff21c463fc267bf77dc89, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:47,085 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689768347084"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689768347084"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689768347084"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689768347084"}]},"ts":"1689768347084"} 2023-07-19 12:05:47,086 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:47,086 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:47,086 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=7b841029b07ea6e492c2c28c4a93a738, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:47,087 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689768347086"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689768347086"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689768347086"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689768347086"}]},"ts":"1689768347086"} 2023-07-19 12:05:47,089 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=27 2023-07-19 12:05:47,090 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=27, state=SUCCESS; OpenRegionProcedure 78e0d689b75ff21c463fc267bf77dc89, server=jenkins-hbase4.apache.org,43697,1689768328415 in 196 msec 2023-07-19 12:05:47,091 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=26 2023-07-19 12:05:47,092 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=t, region=78e0d689b75ff21c463fc267bf77dc89, ASSIGN in 313 msec 2023-07-19 12:05:47,092 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=26, state=SUCCESS; OpenRegionProcedure 7b841029b07ea6e492c2c28c4a93a738, server=jenkins-hbase4.apache.org,43697,1689768328415 in 199 msec 2023-07-19 12:05:47,095 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-19 12:05:47,095 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,39213,1689768323446 after splitting done 2023-07-19 12:05:47,095 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7b841029b07ea6e492c2c28c4a93a738, ASSIGN in 316 msec 2023-07-19 12:05:47,095 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,39213,1689768323446 from processing; numProcessing=0 2023-07-19 12:05:47,098 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,39213,1689768323446, splitWal=true, meta=true in 1.2420 sec 2023-07-19 12:05:47,100 DEBUG [Listener at localhost/34139] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x776125a4 to 127.0.0.1:53047 2023-07-19 12:05:47,100 DEBUG [Listener at localhost/34139] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:47,100 INFO [Listener at localhost/34139] client.TestScannerTimeout(124): END ************ test2772 2023-07-19 12:05:47,113 INFO [Listener at localhost/34139] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 297), OpenFileDescriptor=572 (was 572), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=172 (was 172), ProcessCount=170 (was 170), AvailableMemoryMB=6617 (was 6681) 2023-07-19 12:05:47,113 INFO [Listener at localhost/34139] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-19 12:05:47,114 INFO [Listener at localhost/34139] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-19 12:05:47,114 DEBUG [Listener at localhost/34139] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ad49168 to 127.0.0.1:53047 2023-07-19 12:05:47,114 DEBUG [Listener at localhost/34139] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:47,114 DEBUG [Listener at localhost/34139] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-19 12:05:47,114 DEBUG [Listener at localhost/34139] util.JVMClusterUtil(257): Found active master hash=1365144794, stopped=false 2023-07-19 12:05:47,114 INFO [Listener at localhost/34139] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:47,120 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-19 12:05:47,120 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-19 12:05:47,120 INFO [Listener at localhost/34139] procedure2.ProcedureExecutor(629): Stopping 2023-07-19 12:05:47,120 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:47,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 12:05:47,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 12:05:47,121 DEBUG [Listener at localhost/34139] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b3841ff to 127.0.0.1:53047 2023-07-19 12:05:47,121 DEBUG [Listener at localhost/34139] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:47,121 INFO [Listener at localhost/34139] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,43697,1689768328415' ***** 2023-07-19 12:05:47,121 INFO [Listener at localhost/34139] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-19 12:05:47,122 INFO [RS:2;jenkins-hbase4:43697] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 12:05:47,122 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 12:05:47,122 INFO [RS:2;jenkins-hbase4:43697] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-19 12:05:47,122 INFO [RS:2;jenkins-hbase4:43697] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-19 12:05:47,122 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(3305): Received CLOSE for 78e0d689b75ff21c463fc267bf77dc89 2023-07-19 12:05:47,122 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(3305): Received CLOSE for 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:47,122 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:47,122 DEBUG [RS:2;jenkins-hbase4:43697] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0940959a to 127.0.0.1:53047 2023-07-19 12:05:47,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 78e0d689b75ff21c463fc267bf77dc89, disabling compactions & flushes 2023-07-19 12:05:47,123 DEBUG [RS:2;jenkins-hbase4:43697] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:47,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:47,123 INFO [RS:2;jenkins-hbase4:43697] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 12:05:47,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:47,123 INFO [RS:2;jenkins-hbase4:43697] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 12:05:47,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. after waiting 0 ms 2023-07-19 12:05:47,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:47,123 INFO [RS:2;jenkins-hbase4:43697] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 12:05:47,123 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-19 12:05:47,124 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-19 12:05:47,124 DEBUG [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1478): Online Regions={78e0d689b75ff21c463fc267bf77dc89=t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89., 1588230740=hbase:meta,,1.1588230740, 7b841029b07ea6e492c2c28c4a93a738=hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738.} 2023-07-19 12:05:47,125 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 12:05:47,125 DEBUG [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1504): Waiting on 1588230740, 78e0d689b75ff21c463fc267bf77dc89, 7b841029b07ea6e492c2c28c4a93a738 2023-07-19 12:05:47,125 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 12:05:47,125 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 12:05:47,125 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 12:05:47,125 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 12:05:47,125 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-19 12:05:47,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/default/t/78e0d689b75ff21c463fc267bf77dc89/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-19 12:05:47,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:47,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 78e0d689b75ff21c463fc267bf77dc89: 2023-07-19 12:05:47,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689768326394.78e0d689b75ff21c463fc267bf77dc89. 2023-07-19 12:05:47,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7b841029b07ea6e492c2c28c4a93a738, disabling compactions & flushes 2023-07-19 12:05:47,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:47,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:47,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. after waiting 0 ms 2023-07-19 12:05:47,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:47,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/namespace/7b841029b07ea6e492c2c28c4a93a738/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-19 12:05:47,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:47,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7b841029b07ea6e492c2c28c4a93a738: 2023-07-19 12:05:47,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689768325567.7b841029b07ea6e492c2c28c4a93a738. 2023-07-19 12:05:47,161 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/.tmp/info/58e519590b084bd4a81e72d91075238e 2023-07-19 12:05:47,169 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/.tmp/info/58e519590b084bd4a81e72d91075238e as hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/info/58e519590b084bd4a81e72d91075238e 2023-07-19 12:05:47,177 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/info/58e519590b084bd4a81e72d91075238e, entries=16, sequenceid=24, filesize=6.4 K 2023-07-19 12:05:47,178 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 53ms, sequenceid=24, compaction requested=false 2023-07-19 12:05:47,185 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=17 2023-07-19 12:05:47,186 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 12:05:47,186 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 12:05:47,186 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 12:05:47,186 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-19 12:05:47,325 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,43697,1689768328415; all regions closed. 2023-07-19 12:05:47,332 DEBUG [RS:2;jenkins-hbase4:43697] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs 2023-07-19 12:05:47,332 INFO [RS:2;jenkins-hbase4:43697] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43697%2C1689768328415.meta:.meta(num 1689768346406) 2023-07-19 12:05:47,337 DEBUG [RS:2;jenkins-hbase4:43697] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/oldWALs 2023-07-19 12:05:47,337 INFO [RS:2;jenkins-hbase4:43697] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43697%2C1689768328415:(num 1689768328614) 2023-07-19 12:05:47,337 DEBUG [RS:2;jenkins-hbase4:43697] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:47,337 INFO [RS:2;jenkins-hbase4:43697] regionserver.LeaseManager(133): Closed leases 2023-07-19 12:05:47,337 INFO [RS:2;jenkins-hbase4:43697] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-19 12:05:47,338 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 12:05:47,338 INFO [RS:2;jenkins-hbase4:43697] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43697 2023-07-19 12:05:47,340 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:47,341 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43697,1689768328415 2023-07-19 12:05:47,341 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 12:05:47,341 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43697,1689768328415] 2023-07-19 12:05:47,341 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43697,1689768328415; numProcessing=1 2023-07-19 12:05:47,344 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43697,1689768328415 already deleted, retry=false 2023-07-19 12:05:47,344 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43697,1689768328415 expired; onlineServers=0 2023-07-19 12:05:47,344 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,37041,1689768322202' ***** 2023-07-19 12:05:47,344 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-19 12:05:47,345 DEBUG [M:0;jenkins-hbase4:37041] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10637ee1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 12:05:47,345 INFO [M:0;jenkins-hbase4:37041] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,37041,1689768322202 2023-07-19 12:05:47,345 INFO [M:0;jenkins-hbase4:37041] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,37041,1689768322202; all regions closed. 2023-07-19 12:05:47,345 DEBUG [M:0;jenkins-hbase4:37041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 12:05:47,345 DEBUG [M:0;jenkins-hbase4:37041] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-19 12:05:47,345 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-19 12:05:47,345 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689768324767] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689768324767,5,FailOnTimeoutGroup] 2023-07-19 12:05:47,345 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689768324765] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689768324765,5,FailOnTimeoutGroup] 2023-07-19 12:05:47,345 DEBUG [M:0;jenkins-hbase4:37041] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-19 12:05:47,346 INFO [M:0;jenkins-hbase4:37041] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-19 12:05:47,346 INFO [M:0;jenkins-hbase4:37041] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-19 12:05:47,346 INFO [M:0;jenkins-hbase4:37041] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-19 12:05:47,346 DEBUG [M:0;jenkins-hbase4:37041] master.HMaster(1512): Stopping service threads 2023-07-19 12:05:47,346 INFO [M:0;jenkins-hbase4:37041] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-19 12:05:47,347 ERROR [M:0;jenkins-hbase4:37041] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1324093987) connection to localhost/127.0.0.1:41359 from jenkins,5,PEWorkerGroup] 2023-07-19 12:05:47,347 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-19 12:05:47,347 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 12:05:47,347 INFO [M:0;jenkins-hbase4:37041] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-19 12:05:47,347 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-19 12:05:47,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 12:05:47,348 DEBUG [M:0;jenkins-hbase4:37041] zookeeper.ZKUtil(398): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-19 12:05:47,348 WARN [M:0;jenkins-hbase4:37041] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-19 12:05:47,348 INFO [M:0;jenkins-hbase4:37041] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-19 12:05:47,348 INFO [M:0;jenkins-hbase4:37041] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-19 12:05:47,349 DEBUG [M:0;jenkins-hbase4:37041] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-19 12:05:47,349 INFO [M:0;jenkins-hbase4:37041] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 12:05:47,349 DEBUG [M:0;jenkins-hbase4:37041] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 12:05:47,349 DEBUG [M:0;jenkins-hbase4:37041] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-19 12:05:47,349 DEBUG [M:0;jenkins-hbase4:37041] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 12:05:47,349 INFO [M:0;jenkins-hbase4:37041] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=96.34 KB heapSize=116.35 KB 2023-07-19 12:05:47,365 INFO [M:0;jenkins-hbase4:37041] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=96.34 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/feaddbf2904e4ac8a78de2ec148368ca 2023-07-19 12:05:47,371 DEBUG [M:0;jenkins-hbase4:37041] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/feaddbf2904e4ac8a78de2ec148368ca as hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/feaddbf2904e4ac8a78de2ec148368ca 2023-07-19 12:05:47,377 INFO [M:0;jenkins-hbase4:37041] regionserver.HStore(1080): Added hdfs://localhost:41359/user/jenkins/test-data/dae1866e-059d-4bcb-7aca-81d7e6182b7d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/feaddbf2904e4ac8a78de2ec148368ca, entries=29, sequenceid=225, filesize=8.3 K 2023-07-19 12:05:47,378 INFO [M:0;jenkins-hbase4:37041] regionserver.HRegion(2948): Finished flush of dataSize ~96.34 KB/98654, heapSize ~116.34 KB/119128, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 29ms, sequenceid=225, compaction requested=false 2023-07-19 12:05:47,379 INFO [M:0;jenkins-hbase4:37041] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 12:05:47,379 DEBUG [M:0;jenkins-hbase4:37041] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 12:05:47,383 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 12:05:47,383 INFO [M:0;jenkins-hbase4:37041] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-19 12:05:47,383 INFO [M:0;jenkins-hbase4:37041] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37041 2023-07-19 12:05:47,385 DEBUG [M:0;jenkins-hbase4:37041] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,37041,1689768322202 already deleted, retry=false 2023-07-19 12:05:47,485 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 12:05:47,604 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 12:05:47,604 INFO [M:0;jenkins-hbase4:37041] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,37041,1689768322202; zookeeper connection closed. 2023-07-19 12:05:47,604 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): master:37041-0x1017d78e43d0000, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 12:05:47,704 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 12:05:47,704 INFO [RS:2;jenkins-hbase4:43697] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,43697,1689768328415; zookeeper connection closed. 2023-07-19 12:05:47,704 DEBUG [Listener at localhost/34139-EventThread] zookeeper.ZKWatcher(600): regionserver:43697-0x1017d78e43d0008, quorum=127.0.0.1:53047, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 12:05:47,704 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7f52412c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7f52412c 2023-07-19 12:05:47,705 INFO [Listener at localhost/34139] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-19 12:05:47,705 WARN [Listener at localhost/34139] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-19 12:05:47,708 INFO [Listener at localhost/34139] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 12:05:47,814 WARN [BP-1088350189-172.31.14.131-1689768318955 heartbeating to localhost/127.0.0.1:41359] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-19 12:05:47,814 WARN [BP-1088350189-172.31.14.131-1689768318955 heartbeating to localhost/127.0.0.1:41359] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1088350189-172.31.14.131-1689768318955 (Datanode Uuid e3fc58ad-299d-4544-abc7-36122cc6758c) service to localhost/127.0.0.1:41359 2023-07-19 12:05:47,817 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/dfs/data/data3/current/BP-1088350189-172.31.14.131-1689768318955] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 12:05:47,817 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/dfs/data/data4/current/BP-1088350189-172.31.14.131-1689768318955] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 12:05:47,819 WARN [Listener at localhost/34139] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-19 12:05:47,826 INFO [Listener at localhost/34139] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 12:05:47,932 WARN [BP-1088350189-172.31.14.131-1689768318955 heartbeating to localhost/127.0.0.1:41359] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-19 12:05:47,932 WARN [BP-1088350189-172.31.14.131-1689768318955 heartbeating to localhost/127.0.0.1:41359] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1088350189-172.31.14.131-1689768318955 (Datanode Uuid a72220e0-0099-491d-b9cc-a632d5171137) service to localhost/127.0.0.1:41359 2023-07-19 12:05:47,933 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/dfs/data/data1/current/BP-1088350189-172.31.14.131-1689768318955] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 12:05:47,934 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8fafbd39-8cea-5dd1-5012-06ba9d3b0642/cluster_9db5f6ec-6c8d-38fc-ff77-04dd69dbcfd5/dfs/data/data2/current/BP-1088350189-172.31.14.131-1689768318955] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 12:05:47,971 INFO [Listener at localhost/34139] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 12:05:48,083 INFO [Listener at localhost/34139] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-19 12:05:48,126 INFO [Listener at localhost/34139] hbase.HBaseTestingUtility(1293): Minicluster is down