2023-07-16 09:05:35,967 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-16 09:05:36,192 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50 2023-07-16 09:05:36,204 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-16 09:05:36,205 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f, deleteOnExit=true 2023-07-16 09:05:36,205 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-16 09:05:36,205 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/test.cache.data in system properties and HBase conf 2023-07-16 09:05:36,206 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/hadoop.tmp.dir in system properties and HBase conf 2023-07-16 09:05:36,206 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/hadoop.log.dir in system properties and HBase conf 2023-07-16 09:05:36,207 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-16 09:05:36,207 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-16 09:05:36,207 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-16 09:05:36,316 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-16 09:05:36,683 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-16 09:05:36,687 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-16 09:05:36,687 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-16 09:05:36,688 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-16 09:05:36,688 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-16 09:05:36,688 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-16 09:05:36,689 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-16 09:05:36,689 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-16 09:05:36,689 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-16 09:05:36,689 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-16 09:05:36,690 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/nfs.dump.dir in system properties and HBase conf 2023-07-16 09:05:36,690 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/java.io.tmpdir in system properties and HBase conf 2023-07-16 09:05:36,691 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-16 09:05:36,691 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-16 09:05:36,691 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-16 09:05:37,189 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-16 09:05:37,193 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-16 09:05:37,471 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-16 09:05:37,633 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-16 09:05:37,648 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-16 09:05:37,682 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-16 09:05:37,714 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/java.io.tmpdir/Jetty_localhost_36117_hdfs____x3gdby/webapp 2023-07-16 09:05:37,865 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36117 2023-07-16 09:05:37,875 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-16 09:05:37,875 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-16 09:05:38,281 WARN [Listener at localhost/36827] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-16 09:05:38,348 WARN [Listener at localhost/36827] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-16 09:05:38,366 WARN [Listener at localhost/36827] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-16 09:05:38,373 INFO [Listener at localhost/36827] log.Slf4jLog(67): jetty-6.1.26 2023-07-16 09:05:38,377 INFO [Listener at localhost/36827] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/java.io.tmpdir/Jetty_localhost_42719_datanode____.nbdeew/webapp 2023-07-16 09:05:38,473 INFO [Listener at localhost/36827] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42719 2023-07-16 09:05:38,777 WARN [Listener at localhost/36653] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-16 09:05:38,789 WARN [Listener at localhost/36653] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-16 09:05:38,794 WARN [Listener at localhost/36653] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-16 09:05:38,796 INFO [Listener at localhost/36653] log.Slf4jLog(67): jetty-6.1.26 2023-07-16 09:05:38,801 INFO [Listener at localhost/36653] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/java.io.tmpdir/Jetty_localhost_34073_datanode____pynss2/webapp 2023-07-16 09:05:38,899 INFO [Listener at localhost/36653] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34073 2023-07-16 09:05:38,908 WARN [Listener at localhost/37403] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-16 09:05:39,255 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x82a9785a10374d05: Processing first storage report for DS-1f9749cf-f995-46a7-b446-fc48533aaccf from datanode aa48e06f-2444-4a8e-b26c-1906e3c7be6b 2023-07-16 09:05:39,257 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x82a9785a10374d05: from storage DS-1f9749cf-f995-46a7-b446-fc48533aaccf node DatanodeRegistration(127.0.0.1:42717, datanodeUuid=aa48e06f-2444-4a8e-b26c-1906e3c7be6b, infoPort=39293, infoSecurePort=0, ipcPort=36653, storageInfo=lv=-57;cid=testClusterID;nsid=1926654935;c=1689498337264), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-16 09:05:39,257 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9a1318c3b50e8d9d: Processing first storage report for DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a from datanode 64a861ed-56f4-41e6-abbb-eb71c4f0710a 2023-07-16 09:05:39,257 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9a1318c3b50e8d9d: from storage DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a node DatanodeRegistration(127.0.0.1:41565, datanodeUuid=64a861ed-56f4-41e6-abbb-eb71c4f0710a, infoPort=40911, infoSecurePort=0, ipcPort=37403, storageInfo=lv=-57;cid=testClusterID;nsid=1926654935;c=1689498337264), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-16 09:05:39,257 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x82a9785a10374d05: Processing first storage report for DS-03c19ef5-dff9-4c29-805c-7df91ace5032 from datanode aa48e06f-2444-4a8e-b26c-1906e3c7be6b 2023-07-16 09:05:39,257 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x82a9785a10374d05: from storage DS-03c19ef5-dff9-4c29-805c-7df91ace5032 node DatanodeRegistration(127.0.0.1:42717, datanodeUuid=aa48e06f-2444-4a8e-b26c-1906e3c7be6b, infoPort=39293, infoSecurePort=0, ipcPort=36653, storageInfo=lv=-57;cid=testClusterID;nsid=1926654935;c=1689498337264), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-16 09:05:39,257 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9a1318c3b50e8d9d: Processing first storage report for DS-8a1a483f-177b-49f8-aced-6d1b5adc4f28 from datanode 64a861ed-56f4-41e6-abbb-eb71c4f0710a 2023-07-16 09:05:39,257 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9a1318c3b50e8d9d: from storage DS-8a1a483f-177b-49f8-aced-6d1b5adc4f28 node DatanodeRegistration(127.0.0.1:41565, datanodeUuid=64a861ed-56f4-41e6-abbb-eb71c4f0710a, infoPort=40911, infoSecurePort=0, ipcPort=37403, storageInfo=lv=-57;cid=testClusterID;nsid=1926654935;c=1689498337264), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-16 09:05:39,312 DEBUG [Listener at localhost/37403] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50 2023-07-16 09:05:39,386 INFO [Listener at localhost/37403] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/zookeeper_0, clientPort=53764, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-16 09:05:39,402 INFO [Listener at localhost/37403] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=53764 2023-07-16 09:05:39,412 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:39,415 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:40,065 INFO [Listener at localhost/37403] util.FSUtils(471): Created version file at hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76 with version=8 2023-07-16 09:05:40,066 INFO [Listener at localhost/37403] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/hbase-staging 2023-07-16 09:05:40,373 INFO [Listener at localhost/37403] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-16 09:05:40,830 INFO [Listener at localhost/37403] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-16 09:05:40,862 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:40,862 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:40,862 INFO [Listener at localhost/37403] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-16 09:05:40,863 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:40,863 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-16 09:05:41,000 INFO [Listener at localhost/37403] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-16 09:05:41,070 DEBUG [Listener at localhost/37403] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-16 09:05:41,167 INFO [Listener at localhost/37403] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46859 2023-07-16 09:05:41,178 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:41,180 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:41,200 INFO [Listener at localhost/37403] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46859 connecting to ZooKeeper ensemble=127.0.0.1:53764 2023-07-16 09:05:41,241 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:468590x0, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-16 09:05:41,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:46859-0x1016d614b290000 connected 2023-07-16 09:05:41,270 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-16 09:05:41,271 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-16 09:05:41,274 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-16 09:05:41,281 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46859 2023-07-16 09:05:41,282 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46859 2023-07-16 09:05:41,282 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46859 2023-07-16 09:05:41,283 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46859 2023-07-16 09:05:41,283 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46859 2023-07-16 09:05:41,289 INFO [Listener at localhost/37403] master.HMaster(444): hbase.rootdir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76, hbase.cluster.distributed=false 2023-07-16 09:05:41,358 INFO [Listener at localhost/37403] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-16 09:05:41,359 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:41,359 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:41,359 INFO [Listener at localhost/37403] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-16 09:05:41,359 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:41,359 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-16 09:05:41,364 INFO [Listener at localhost/37403] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-16 09:05:41,367 INFO [Listener at localhost/37403] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41705 2023-07-16 09:05:41,369 INFO [Listener at localhost/37403] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-16 09:05:41,375 DEBUG [Listener at localhost/37403] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-16 09:05:41,376 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:41,378 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:41,380 INFO [Listener at localhost/37403] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41705 connecting to ZooKeeper ensemble=127.0.0.1:53764 2023-07-16 09:05:41,384 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:417050x0, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-16 09:05:41,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:41705-0x1016d614b290001 connected 2023-07-16 09:05:41,385 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-16 09:05:41,386 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-16 09:05:41,387 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-16 09:05:41,387 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41705 2023-07-16 09:05:41,387 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41705 2023-07-16 09:05:41,388 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41705 2023-07-16 09:05:41,389 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41705 2023-07-16 09:05:41,389 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41705 2023-07-16 09:05:41,403 INFO [Listener at localhost/37403] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-16 09:05:41,403 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:41,403 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:41,403 INFO [Listener at localhost/37403] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-16 09:05:41,403 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:41,404 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-16 09:05:41,404 INFO [Listener at localhost/37403] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-16 09:05:41,405 INFO [Listener at localhost/37403] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43645 2023-07-16 09:05:41,406 INFO [Listener at localhost/37403] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-16 09:05:41,406 DEBUG [Listener at localhost/37403] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-16 09:05:41,407 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:41,410 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:41,412 INFO [Listener at localhost/37403] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43645 connecting to ZooKeeper ensemble=127.0.0.1:53764 2023-07-16 09:05:41,417 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:436450x0, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-16 09:05:41,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43645-0x1016d614b290002 connected 2023-07-16 09:05:41,418 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-16 09:05:41,419 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-16 09:05:41,420 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-16 09:05:41,420 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43645 2023-07-16 09:05:41,420 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43645 2023-07-16 09:05:41,421 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43645 2023-07-16 09:05:41,421 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43645 2023-07-16 09:05:41,422 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43645 2023-07-16 09:05:41,423 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:05:41,432 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-16 09:05:41,433 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:05:41,453 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-16 09:05:41,453 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-16 09:05:41,453 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-16 09:05:41,453 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:05:41,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-16 09:05:41,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46859,1689498340206 from backup master directory 2023-07-16 09:05:41,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-16 09:05:41,459 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:05:41,459 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-16 09:05:41,459 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-16 09:05:41,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:05:41,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-16 09:05:41,464 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-16 09:05:41,545 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/hbase.id with ID: 1975af79-dd1e-4db1-8abc-56f2cdde95d7 2023-07-16 09:05:41,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:41,602 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:05:41,644 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x636dce05 to 127.0.0.1:53764 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-16 09:05:41,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@377fe0e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-16 09:05:41,700 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-16 09:05:41,702 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-16 09:05:41,718 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-16 09:05:41,719 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-16 09:05:41,720 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-16 09:05:41,724 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-16 09:05:41,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-16 09:05:41,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/data/master/store-tmp 2023-07-16 09:05:41,788 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:41,788 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-16 09:05:41,788 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-16 09:05:41,788 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-16 09:05:41,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-16 09:05:41,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-16 09:05:41,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-16 09:05:41,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-16 09:05:41,790 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/WALs/jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:05:41,812 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46859%2C1689498340206, suffix=, logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/WALs/jenkins-hbase4.apache.org,46859,1689498340206, archiveDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/oldWALs, maxLogs=10 2023-07-16 09:05:41,865 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK] 2023-07-16 09:05:41,865 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK] 2023-07-16 09:05:41,873 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-16 09:05:41,937 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/WALs/jenkins-hbase4.apache.org,46859,1689498340206/jenkins-hbase4.apache.org%2C46859%2C1689498340206.1689498341822 2023-07-16 09:05:41,937 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK], DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK]] 2023-07-16 09:05:41,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:05:41,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:41,941 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-16 09:05:41,942 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-16 09:05:41,997 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-16 09:05:42,004 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-16 09:05:42,027 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-16 09:05:42,038 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:42,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-16 09:05:42,046 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-16 09:05:42,060 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-16 09:05:42,065 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-16 09:05:42,066 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71379375, jitterRate=0.06363557279109955}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-16 09:05:42,066 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-16 09:05:42,067 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-16 09:05:42,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-16 09:05:42,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-16 09:05:42,093 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-16 09:05:42,095 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-16 09:05:42,127 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-16 09:05:42,127 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-16 09:05:42,152 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-16 09:05:42,157 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-16 09:05:42,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-16 09:05:42,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-16 09:05:42,186 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-16 09:05:42,191 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-16 09:05:42,195 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-16 09:05:42,198 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:05:42,200 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-16 09:05:42,200 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-16 09:05:42,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-16 09:05:42,216 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-16 09:05:42,216 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-16 09:05:42,216 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:05:42,216 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-16 09:05:42,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,46859,1689498340206, sessionid=0x1016d614b290000, setting cluster-up flag (Was=false) 2023-07-16 09:05:42,231 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:05:42,237 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-16 09:05:42,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:05:42,244 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:05:42,249 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-16 09:05:42,251 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:05:42,253 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.hbase-snapshot/.tmp 2023-07-16 09:05:42,325 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(951): ClusterId : 1975af79-dd1e-4db1-8abc-56f2cdde95d7 2023-07-16 09:05:42,325 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(951): ClusterId : 1975af79-dd1e-4db1-8abc-56f2cdde95d7 2023-07-16 09:05:42,330 DEBUG [RS:0;jenkins-hbase4:41705] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-16 09:05:42,330 DEBUG [RS:1;jenkins-hbase4:43645] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-16 09:05:42,337 DEBUG [RS:0;jenkins-hbase4:41705] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-16 09:05:42,337 DEBUG [RS:1;jenkins-hbase4:43645] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-16 09:05:42,337 DEBUG [RS:0;jenkins-hbase4:41705] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-16 09:05:42,337 DEBUG [RS:1;jenkins-hbase4:43645] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-16 09:05:42,341 DEBUG [RS:0;jenkins-hbase4:41705] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-16 09:05:42,342 DEBUG [RS:1;jenkins-hbase4:43645] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-16 09:05:42,343 DEBUG [RS:1;jenkins-hbase4:43645] zookeeper.ReadOnlyZKClient(139): Connect 0x11caf6b1 to 127.0.0.1:53764 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-16 09:05:42,343 DEBUG [RS:0;jenkins-hbase4:41705] zookeeper.ReadOnlyZKClient(139): Connect 0x4c967409 to 127.0.0.1:53764 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-16 09:05:42,351 DEBUG [RS:1;jenkins-hbase4:43645] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f0d0af8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-16 09:05:42,351 DEBUG [RS:0;jenkins-hbase4:41705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@425193eb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-16 09:05:42,351 DEBUG [RS:1;jenkins-hbase4:43645] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b329dc1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-16 09:05:42,351 DEBUG [RS:0;jenkins-hbase4:41705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@466e1f79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-16 09:05:42,374 DEBUG [RS:0;jenkins-hbase4:41705] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:41705 2023-07-16 09:05:42,375 DEBUG [RS:1;jenkins-hbase4:43645] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:43645 2023-07-16 09:05:42,380 INFO [RS:1;jenkins-hbase4:43645] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-16 09:05:42,380 INFO [RS:1;jenkins-hbase4:43645] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-16 09:05:42,380 INFO [RS:0;jenkins-hbase4:41705] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-16 09:05:42,380 DEBUG [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1022): About to register with Master. 2023-07-16 09:05:42,381 INFO [RS:0;jenkins-hbase4:41705] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-16 09:05:42,381 DEBUG [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1022): About to register with Master. 2023-07-16 09:05:42,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-16 09:05:42,384 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,46859,1689498340206 with isa=jenkins-hbase4.apache.org/172.31.14.131:43645, startcode=1689498341402 2023-07-16 09:05:42,384 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,46859,1689498340206 with isa=jenkins-hbase4.apache.org/172.31.14.131:41705, startcode=1689498341358 2023-07-16 09:05:42,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-16 09:05:42,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-16 09:05:42,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-16 09:05:42,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-16 09:05:42,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-16 09:05:42,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,393 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-16 09:05:42,393 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,394 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689498372394 2023-07-16 09:05:42,396 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-16 09:05:42,400 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-16 09:05:42,401 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-16 09:05:42,404 DEBUG [RS:0;jenkins-hbase4:41705] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-16 09:05:42,404 DEBUG [RS:1;jenkins-hbase4:43645] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-16 09:05:42,407 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-16 09:05:42,407 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-16 09:05:42,414 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-16 09:05:42,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-16 09:05:42,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-16 09:05:42,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-16 09:05:42,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,417 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-16 09:05:42,419 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-16 09:05:42,419 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-16 09:05:42,421 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-16 09:05:42,421 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-16 09:05:42,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689498342424,5,FailOnTimeoutGroup] 2023-07-16 09:05:42,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689498342424,5,FailOnTimeoutGroup] 2023-07-16 09:05:42,424 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,425 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-16 09:05:42,426 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,426 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,446 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-16 09:05:42,449 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-16 09:05:42,449 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76 2023-07-16 09:05:42,472 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50151, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-16 09:05:42,472 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50057, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-16 09:05:42,476 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:42,480 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-16 09:05:42,482 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info 2023-07-16 09:05:42,483 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-16 09:05:42,484 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:42,484 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-16 09:05:42,486 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/rep_barrier 2023-07-16 09:05:42,487 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46859] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:42,488 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-16 09:05:42,489 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:42,489 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-16 09:05:42,491 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table 2023-07-16 09:05:42,491 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-16 09:05:42,492 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:42,494 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740 2023-07-16 09:05:42,494 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740 2023-07-16 09:05:42,498 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-16 09:05:42,500 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-16 09:05:42,501 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46859] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:42,505 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-16 09:05:42,506 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69908771, jitterRate=0.04172186553478241}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-16 09:05:42,506 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-16 09:05:42,506 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-16 09:05:42,506 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-16 09:05:42,506 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-16 09:05:42,506 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-16 09:05:42,506 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-16 09:05:42,508 DEBUG [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76 2023-07-16 09:05:42,508 DEBUG [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76 2023-07-16 09:05:42,508 DEBUG [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:36827 2023-07-16 09:05:42,508 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-16 09:05:42,508 DEBUG [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:36827 2023-07-16 09:05:42,508 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-16 09:05:42,508 DEBUG [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-16 09:05:42,508 DEBUG [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-16 09:05:42,513 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-16 09:05:42,513 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-16 09:05:42,517 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:05:42,517 DEBUG [RS:0;jenkins-hbase4:41705] zookeeper.ZKUtil(162): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:42,517 DEBUG [RS:1;jenkins-hbase4:43645] zookeeper.ZKUtil(162): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:42,517 WARN [RS:0;jenkins-hbase4:41705] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-16 09:05:42,518 WARN [RS:1;jenkins-hbase4:43645] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-16 09:05:42,518 INFO [RS:0;jenkins-hbase4:41705] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-16 09:05:42,519 INFO [RS:1;jenkins-hbase4:43645] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-16 09:05:42,519 DEBUG [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1948): logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:42,519 DEBUG [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1948): logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:42,520 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41705,1689498341358] 2023-07-16 09:05:42,520 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43645,1689498341402] 2023-07-16 09:05:42,527 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-16 09:05:42,531 DEBUG [RS:0;jenkins-hbase4:41705] zookeeper.ZKUtil(162): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:42,531 DEBUG [RS:1;jenkins-hbase4:43645] zookeeper.ZKUtil(162): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:42,531 DEBUG [RS:0;jenkins-hbase4:41705] zookeeper.ZKUtil(162): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:42,531 DEBUG [RS:1;jenkins-hbase4:43645] zookeeper.ZKUtil(162): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:42,542 DEBUG [RS:0;jenkins-hbase4:41705] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-16 09:05:42,542 DEBUG [RS:1;jenkins-hbase4:43645] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-16 09:05:42,544 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-16 09:05:42,546 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-16 09:05:42,552 INFO [RS:1;jenkins-hbase4:43645] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-16 09:05:42,552 INFO [RS:0;jenkins-hbase4:41705] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-16 09:05:42,574 INFO [RS:1;jenkins-hbase4:43645] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-16 09:05:42,574 INFO [RS:0;jenkins-hbase4:41705] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-16 09:05:42,578 INFO [RS:1;jenkins-hbase4:43645] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-16 09:05:42,578 INFO [RS:0;jenkins-hbase4:41705] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-16 09:05:42,578 INFO [RS:1;jenkins-hbase4:43645] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,578 INFO [RS:0;jenkins-hbase4:41705] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,579 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-16 09:05:42,579 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-16 09:05:42,586 INFO [RS:1;jenkins-hbase4:43645] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,586 INFO [RS:0;jenkins-hbase4:41705] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,587 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,587 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,587 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,587 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,587 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,587 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,587 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,587 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,588 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,588 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,588 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-16 09:05:42,588 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-16 09:05:42,588 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,588 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,588 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,588 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,588 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,588 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,588 DEBUG [RS:1;jenkins-hbase4:43645] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,589 DEBUG [RS:0;jenkins-hbase4:41705] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:42,592 INFO [RS:1;jenkins-hbase4:43645] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,593 INFO [RS:0;jenkins-hbase4:41705] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,593 INFO [RS:1;jenkins-hbase4:43645] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,593 INFO [RS:0;jenkins-hbase4:41705] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,593 INFO [RS:1;jenkins-hbase4:43645] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,593 INFO [RS:0;jenkins-hbase4:41705] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,611 INFO [RS:1;jenkins-hbase4:43645] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-16 09:05:42,611 INFO [RS:0;jenkins-hbase4:41705] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-16 09:05:42,614 INFO [RS:1;jenkins-hbase4:43645] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43645,1689498341402-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,614 INFO [RS:0;jenkins-hbase4:41705] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41705,1689498341358-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:42,634 INFO [RS:0;jenkins-hbase4:41705] regionserver.Replication(203): jenkins-hbase4.apache.org,41705,1689498341358 started 2023-07-16 09:05:42,634 INFO [RS:1;jenkins-hbase4:43645] regionserver.Replication(203): jenkins-hbase4.apache.org,43645,1689498341402 started 2023-07-16 09:05:42,634 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,41705,1689498341358, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41705, sessionid=0x1016d614b290001 2023-07-16 09:05:42,634 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,43645,1689498341402, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43645, sessionid=0x1016d614b290002 2023-07-16 09:05:42,634 DEBUG [RS:0;jenkins-hbase4:41705] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-16 09:05:42,634 DEBUG [RS:1;jenkins-hbase4:43645] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-16 09:05:42,634 DEBUG [RS:0;jenkins-hbase4:41705] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:42,634 DEBUG [RS:1;jenkins-hbase4:43645] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:42,635 DEBUG [RS:0;jenkins-hbase4:41705] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41705,1689498341358' 2023-07-16 09:05:42,635 DEBUG [RS:1;jenkins-hbase4:43645] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43645,1689498341402' 2023-07-16 09:05:42,636 DEBUG [RS:1;jenkins-hbase4:43645] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-16 09:05:42,635 DEBUG [RS:0;jenkins-hbase4:41705] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-16 09:05:42,636 DEBUG [RS:0;jenkins-hbase4:41705] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-16 09:05:42,636 DEBUG [RS:1;jenkins-hbase4:43645] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-16 09:05:42,637 DEBUG [RS:0;jenkins-hbase4:41705] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-16 09:05:42,637 DEBUG [RS:0;jenkins-hbase4:41705] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-16 09:05:42,637 DEBUG [RS:1;jenkins-hbase4:43645] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-16 09:05:42,637 DEBUG [RS:0;jenkins-hbase4:41705] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:42,637 DEBUG [RS:1;jenkins-hbase4:43645] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-16 09:05:42,637 DEBUG [RS:0;jenkins-hbase4:41705] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41705,1689498341358' 2023-07-16 09:05:42,638 DEBUG [RS:0;jenkins-hbase4:41705] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-16 09:05:42,637 DEBUG [RS:1;jenkins-hbase4:43645] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:42,638 DEBUG [RS:1;jenkins-hbase4:43645] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43645,1689498341402' 2023-07-16 09:05:42,638 DEBUG [RS:1;jenkins-hbase4:43645] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-16 09:05:42,638 DEBUG [RS:0;jenkins-hbase4:41705] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-16 09:05:42,638 DEBUG [RS:1;jenkins-hbase4:43645] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-16 09:05:42,639 DEBUG [RS:0;jenkins-hbase4:41705] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-16 09:05:42,639 INFO [RS:0;jenkins-hbase4:41705] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-16 09:05:42,639 DEBUG [RS:1;jenkins-hbase4:43645] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-16 09:05:42,639 INFO [RS:0;jenkins-hbase4:41705] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-16 09:05:42,639 INFO [RS:1;jenkins-hbase4:43645] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-16 09:05:42,639 INFO [RS:1;jenkins-hbase4:43645] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-16 09:05:42,698 DEBUG [jenkins-hbase4:46859] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-16 09:05:42,704 DEBUG [jenkins-hbase4:46859] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-16 09:05:42,710 DEBUG [jenkins-hbase4:46859] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-16 09:05:42,710 DEBUG [jenkins-hbase4:46859] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-16 09:05:42,710 DEBUG [jenkins-hbase4:46859] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-16 09:05:42,713 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41705,1689498341358, state=OPENING 2023-07-16 09:05:42,720 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-16 09:05:42,722 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:05:42,722 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-16 09:05:42,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41705,1689498341358}] 2023-07-16 09:05:42,750 INFO [RS:1;jenkins-hbase4:43645] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43645%2C1689498341402, suffix=, logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402, archiveDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs, maxLogs=32 2023-07-16 09:05:42,750 INFO [RS:0;jenkins-hbase4:41705] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41705%2C1689498341358, suffix=, logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358, archiveDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs, maxLogs=32 2023-07-16 09:05:42,772 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK] 2023-07-16 09:05:42,772 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK] 2023-07-16 09:05:42,778 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK] 2023-07-16 09:05:42,778 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK] 2023-07-16 09:05:42,780 INFO [RS:1;jenkins-hbase4:43645] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754 2023-07-16 09:05:42,781 DEBUG [RS:1;jenkins-hbase4:43645] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK], DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK]] 2023-07-16 09:05:42,784 INFO [RS:0;jenkins-hbase4:41705] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754 2023-07-16 09:05:42,784 DEBUG [RS:0;jenkins-hbase4:41705] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK], DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK]] 2023-07-16 09:05:42,911 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:42,913 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-16 09:05:42,916 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49610, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-16 09:05:42,928 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-16 09:05:42,928 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-16 09:05:42,932 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta, suffix=.meta, logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358, archiveDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs, maxLogs=32 2023-07-16 09:05:42,948 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK] 2023-07-16 09:05:42,949 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK] 2023-07-16 09:05:42,955 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta 2023-07-16 09:05:42,957 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK], DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK]] 2023-07-16 09:05:42,958 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:05:42,960 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-16 09:05:42,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-16 09:05:43,026 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-16 09:05:43,031 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-16 09:05:43,031 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:43,032 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-16 09:05:43,032 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-16 09:05:43,035 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-16 09:05:43,037 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info 2023-07-16 09:05:43,037 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info 2023-07-16 09:05:43,037 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-16 09:05:43,038 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:43,038 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-16 09:05:43,040 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/rep_barrier 2023-07-16 09:05:43,040 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/rep_barrier 2023-07-16 09:05:43,040 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-16 09:05:43,041 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:43,041 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-16 09:05:43,043 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table 2023-07-16 09:05:43,043 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table 2023-07-16 09:05:43,043 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-16 09:05:43,044 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:43,046 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740 2023-07-16 09:05:43,048 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740 2023-07-16 09:05:43,051 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-16 09:05:43,054 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-16 09:05:43,055 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60292173, jitterRate=-0.10157661139965057}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-16 09:05:43,055 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-16 09:05:43,064 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689498342901 2023-07-16 09:05:43,081 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-16 09:05:43,081 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-16 09:05:43,082 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41705,1689498341358, state=OPEN 2023-07-16 09:05:43,085 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-16 09:05:43,086 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-16 09:05:43,089 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-16 09:05:43,089 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41705,1689498341358 in 360 msec 2023-07-16 09:05:43,094 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-16 09:05:43,094 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 563 msec 2023-07-16 09:05:43,099 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 790 msec 2023-07-16 09:05:43,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689498343099, completionTime=-1 2023-07-16 09:05:43,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-16 09:05:43,099 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-16 09:05:43,163 DEBUG [hconnection-0xad5928b-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:05:43,167 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49614, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:05:43,183 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-16 09:05:43,183 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689498403183 2023-07-16 09:05:43,183 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689498463183 2023-07-16 09:05:43,183 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 83 msec 2023-07-16 09:05:43,205 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46859,1689498340206-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:43,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46859,1689498340206-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:43,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46859,1689498340206-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:43,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46859, period=300000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:43,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:43,214 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-16 09:05:43,223 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-16 09:05:43,225 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-16 09:05:43,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-16 09:05:43,237 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-16 09:05:43,239 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-16 09:05:43,260 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:43,263 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2 empty. 2023-07-16 09:05:43,264 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:43,264 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-16 09:05:43,292 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-16 09:05:43,294 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7ba9ad3beeb428e3628842526c2eadb2, NAME => 'hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp 2023-07-16 09:05:43,310 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:43,310 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 7ba9ad3beeb428e3628842526c2eadb2, disabling compactions & flushes 2023-07-16 09:05:43,310 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:43,310 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:43,310 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. after waiting 0 ms 2023-07-16 09:05:43,310 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:43,310 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:43,310 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 7ba9ad3beeb428e3628842526c2eadb2: 2023-07-16 09:05:43,315 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-16 09:05:43,330 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689498343317"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689498343317"}]},"ts":"1689498343317"} 2023-07-16 09:05:43,360 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-16 09:05:43,361 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-16 09:05:43,366 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689498343362"}]},"ts":"1689498343362"} 2023-07-16 09:05:43,370 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-16 09:05:43,375 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-16 09:05:43,377 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-16 09:05:43,377 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-16 09:05:43,377 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-16 09:05:43,380 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN}] 2023-07-16 09:05:43,383 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN 2023-07-16 09:05:43,385 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41705,1689498341358; forceNewPlan=false, retain=false 2023-07-16 09:05:43,538 INFO [jenkins-hbase4:46859] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-16 09:05:43,539 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7ba9ad3beeb428e3628842526c2eadb2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:43,539 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689498343538"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689498343538"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689498343538"}]},"ts":"1689498343538"} 2023-07-16 09:05:43,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 7ba9ad3beeb428e3628842526c2eadb2, server=jenkins-hbase4.apache.org,41705,1689498341358}] 2023-07-16 09:05:43,704 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:43,705 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7ba9ad3beeb428e3628842526c2eadb2, NAME => 'hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:05:43,706 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:43,706 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:43,706 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:43,706 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:43,708 INFO [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:43,710 DEBUG [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/info 2023-07-16 09:05:43,710 DEBUG [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/info 2023-07-16 09:05:43,711 INFO [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7ba9ad3beeb428e3628842526c2eadb2 columnFamilyName info 2023-07-16 09:05:43,712 INFO [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] regionserver.HStore(310): Store=7ba9ad3beeb428e3628842526c2eadb2/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:43,713 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:43,714 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:43,719 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:43,722 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-16 09:05:43,723 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7ba9ad3beeb428e3628842526c2eadb2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68824000, jitterRate=0.025557518005371094}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-16 09:05:43,723 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7ba9ad3beeb428e3628842526c2eadb2: 2023-07-16 09:05:43,725 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2., pid=6, masterSystemTime=1689498343697 2023-07-16 09:05:43,729 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:43,729 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:43,730 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7ba9ad3beeb428e3628842526c2eadb2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:43,731 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689498343730"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689498343730"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689498343730"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689498343730"}]},"ts":"1689498343730"} 2023-07-16 09:05:43,738 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-16 09:05:43,738 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 7ba9ad3beeb428e3628842526c2eadb2, server=jenkins-hbase4.apache.org,41705,1689498341358 in 190 msec 2023-07-16 09:05:43,742 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-16 09:05:43,742 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN in 358 msec 2023-07-16 09:05:43,744 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-16 09:05:43,744 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689498343744"}]},"ts":"1689498343744"} 2023-07-16 09:05:43,747 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-16 09:05:43,751 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-16 09:05:43,753 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 525 msec 2023-07-16 09:05:43,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-16 09:05:43,840 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-16 09:05:43,840 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:05:43,877 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-16 09:05:43,899 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-16 09:05:43,905 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 39 msec 2023-07-16 09:05:43,914 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-16 09:05:43,924 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-16 09:05:43,929 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-07-16 09:05:43,940 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-16 09:05:43,942 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-16 09:05:43,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.484sec 2023-07-16 09:05:43,947 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-16 09:05:43,948 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-16 09:05:43,948 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-16 09:05:43,949 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46859,1689498340206-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-16 09:05:43,950 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46859,1689498340206-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-16 09:05:43,959 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-16 09:05:44,031 DEBUG [Listener at localhost/37403] zookeeper.ReadOnlyZKClient(139): Connect 0x61901cae to 127.0.0.1:53764 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-16 09:05:44,036 DEBUG [Listener at localhost/37403] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c77a3bc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-16 09:05:44,049 DEBUG [hconnection-0x7bfc2c0d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:05:44,061 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49628, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:05:44,071 INFO [Listener at localhost/37403] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:05:44,080 DEBUG [Listener at localhost/37403] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-16 09:05:44,082 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52916, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-16 09:05:44,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-16 09:05:44,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-16 09:05:44,099 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-16 09:05:44,101 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-16 09:05:44,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-16 09:05:44,104 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,105 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d empty. 2023-07-16 09:05:44,107 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,107 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-16 09:05:44,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-16 09:05:44,125 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-16 09:05:44,126 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => f14b930b48c75cc2e0ecf1557d0ca60d, NAME => 't,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/.tmp 2023-07-16 09:05:44,141 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:44,141 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing f14b930b48c75cc2e0ecf1557d0ca60d, disabling compactions & flushes 2023-07-16 09:05:44,141 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,141 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,141 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. after waiting 0 ms 2023-07-16 09:05:44,141 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,141 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,141 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for f14b930b48c75cc2e0ecf1557d0ca60d: 2023-07-16 09:05:44,145 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-16 09:05:44,146 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689498344146"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689498344146"}]},"ts":"1689498344146"} 2023-07-16 09:05:44,149 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-16 09:05:44,150 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-16 09:05:44,150 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689498344150"}]},"ts":"1689498344150"} 2023-07-16 09:05:44,153 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-16 09:05:44,158 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-16 09:05:44,159 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-16 09:05:44,159 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-16 09:05:44,159 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-16 09:05:44,159 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN}] 2023-07-16 09:05:44,161 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN 2023-07-16 09:05:44,162 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41705,1689498341358; forceNewPlan=false, retain=false 2023-07-16 09:05:44,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-16 09:05:44,312 INFO [jenkins-hbase4:46859] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-16 09:05:44,313 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f14b930b48c75cc2e0ecf1557d0ca60d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:44,313 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689498344313"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689498344313"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689498344313"}]},"ts":"1689498344313"} 2023-07-16 09:05:44,317 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f14b930b48c75cc2e0ecf1557d0ca60d, server=jenkins-hbase4.apache.org,41705,1689498341358}] 2023-07-16 09:05:44,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-16 09:05:44,476 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,476 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f14b930b48c75cc2e0ecf1557d0ca60d, NAME => 't,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:05:44,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:44,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,479 INFO [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,481 DEBUG [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/f 2023-07-16 09:05:44,481 DEBUG [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/f 2023-07-16 09:05:44,482 INFO [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f14b930b48c75cc2e0ecf1557d0ca60d columnFamilyName f 2023-07-16 09:05:44,483 INFO [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] regionserver.HStore(310): Store=f14b930b48c75cc2e0ecf1557d0ca60d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:44,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-16 09:05:44,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f14b930b48c75cc2e0ecf1557d0ca60d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73848503, jitterRate=0.10042844712734222}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-16 09:05:44,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f14b930b48c75cc2e0ecf1557d0ca60d: 2023-07-16 09:05:44,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d., pid=11, masterSystemTime=1689498344470 2023-07-16 09:05:44,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,498 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,499 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f14b930b48c75cc2e0ecf1557d0ca60d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:44,499 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689498344499"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689498344499"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689498344499"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689498344499"}]},"ts":"1689498344499"} 2023-07-16 09:05:44,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-16 09:05:44,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f14b930b48c75cc2e0ecf1557d0ca60d, server=jenkins-hbase4.apache.org,41705,1689498341358 in 185 msec 2023-07-16 09:05:44,508 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-16 09:05:44,508 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN in 346 msec 2023-07-16 09:05:44,509 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-16 09:05:44,509 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689498344509"}]},"ts":"1689498344509"} 2023-07-16 09:05:44,511 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-16 09:05:44,514 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-16 09:05:44,517 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 419 msec 2023-07-16 09:05:44,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-16 09:05:44,722 INFO [Listener at localhost/37403] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-16 09:05:44,723 DEBUG [Listener at localhost/37403] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-16 09:05:44,724 INFO [Listener at localhost/37403] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-16 09:05:44,728 INFO [Listener at localhost/37403] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-16 09:05:44,729 INFO [Listener at localhost/37403] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-16 09:05:44,729 INFO [Listener at localhost/37403] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-16 09:05:44,775 INFO [Listener at localhost/37403] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=581, MaxFileDescriptor=60000, SystemLoadAverage=216, ProcessCount=169, AvailableMemoryMB=5862 2023-07-16 09:05:44,777 INFO [Listener at localhost/37403] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-16 09:05:44,780 DEBUG [Listener at localhost/37403] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-16 09:05:44,781 DEBUG [Listener at localhost/37403] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,782 INFO [Listener at localhost/37403] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-16 09:05:44,782 INFO [Listener at localhost/37403] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-16 09:05:44,785 INFO [Listener at localhost/37403] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689498343744/Put/vlen=2/seqid=0} 2023-07-16 09:05:44,785 INFO [Listener at localhost/37403] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-16 09:05:44,785 INFO [Listener at localhost/37403] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2./info:regioninfo/1689498343730/Put/vlen=41/seqid=0, hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2./info:seqnumDuringOpen/1689498343730/Put/vlen=8/seqid=0, hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2./info:server/1689498343730/Put/vlen=31/seqid=0, hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2./info:serverstartcode/1689498343730/Put/vlen=8/seqid=0, hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2./info:sn/1689498343538/Put/vlen=45/seqid=0, hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2./info:state/1689498343730/Put/vlen=4/seqid=0} 2023-07-16 09:05:44,786 INFO [Listener at localhost/37403] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 7ba9ad3beeb428e3628842526c2eadb2, NAME => 'hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:05:44,786 INFO [Listener at localhost/37403] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689498344509/Put/vlen=2/seqid=0} 2023-07-16 09:05:44,786 INFO [Listener at localhost/37403] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-16 09:05:44,786 INFO [Listener at localhost/37403] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d./info:regioninfo/1689498344499/Put/vlen=35/seqid=0, t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d./info:seqnumDuringOpen/1689498344499/Put/vlen=8/seqid=0, t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d./info:server/1689498344499/Put/vlen=31/seqid=0, t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d./info:serverstartcode/1689498344499/Put/vlen=8/seqid=0, t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d./info:sn/1689498344313/Put/vlen=45/seqid=0, t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d./info:state/1689498344499/Put/vlen=4/seqid=0} 2023-07-16 09:05:44,786 INFO [Listener at localhost/37403] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => f14b930b48c75cc2e0ecf1557d0ca60d, NAME => 't,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:05:44,787 DEBUG [Listener at localhost/37403] zookeeper.ReadOnlyZKClient(139): Connect 0x3e82be35 to 127.0.0.1:53764 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-16 09:05:44,791 DEBUG [Listener at localhost/37403] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@192ca9e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-16 09:05:44,792 INFO [Listener at localhost/37403] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-16 09:05:44,792 INFO [Listener at localhost/37403] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-16 09:05:44,794 DEBUG [hconnection-0x4953d7ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:05:44,797 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:05:44,802 INFO [Listener at localhost/37403] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-16 09:05:44,805 ERROR [Listener at localhost/37403] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,41705,1689498341358: die! ***** 2023-07-16 09:05:44,806 ERROR [Listener at localhost/37403] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-16 09:05:44,809 DEBUG [Listener at localhost/37403] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-16 09:05:44,812 DEBUG [Listener at localhost/37403] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-16 09:05:44,812 DEBUG [Listener at localhost/37403] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-16 09:05:44,813 DEBUG [Listener at localhost/37403] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-16 09:05:44,813 INFO [Listener at localhost/37403] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 884473856, "init": 513802240, "max": 2051014656, "used": 386810208 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 111049808 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-16 09:05:44,817 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46859] master.MasterRpcServices(612): jenkins-hbase4.apache.org,41705,1689498341358 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,41705,1689498341358: die! ***** 2023-07-16 09:05:44,819 INFO [Listener at localhost/37403] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,41705,1689498341358' ***** 2023-07-16 09:05:44,819 INFO [Listener at localhost/37403] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-16 09:05:44,820 INFO [RS:0;jenkins-hbase4:41705] regionserver.HeapMemoryManager(220): Stopping 2023-07-16 09:05:44,820 INFO [RS:0;jenkins-hbase4:41705] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-16 09:05:44,820 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-16 09:05:44,820 INFO [RS:0;jenkins-hbase4:41705] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-16 09:05:44,820 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(3305): Received CLOSE for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:44,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41705] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 2625807966707646476 2023-07-16 09:05:44,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41705] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41705,1689498341358 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-16 09:05:44,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41705] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:49640 deadline: 1689498404820, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41705,1689498341358 aborting 2023-07-16 09:05:44,821 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(3305): Received CLOSE for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:44,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f14b930b48c75cc2e0ecf1557d0ca60d, disabling compactions & flushes 2023-07-16 09:05:44,822 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:44,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,822 DEBUG [RS:0;jenkins-hbase4:41705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c967409 to 127.0.0.1:53764 2023-07-16 09:05:44,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,823 DEBUG [RS:0;jenkins-hbase4:41705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:05:44,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. after waiting 0 ms 2023-07-16 09:05:44,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,823 INFO [RS:0;jenkins-hbase4:41705] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-16 09:05:44,823 INFO [RS:0;jenkins-hbase4:41705] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-16 09:05:44,823 INFO [RS:0;jenkins-hbase4:41705] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-16 09:05:44,823 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-16 09:05:44,823 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,824 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-16 09:05:44,825 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-16 09:05:44,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f14b930b48c75cc2e0ecf1557d0ca60d: 2023-07-16 09:05:44,825 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-16 09:05:44,825 DEBUG [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, f14b930b48c75cc2e0ecf1557d0ca60d=t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d., 7ba9ad3beeb428e3628842526c2eadb2=hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.} 2023-07-16 09:05:44,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:44,825 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-16 09:05:44,825 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-16 09:05:44,825 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-16 09:05:44,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7ba9ad3beeb428e3628842526c2eadb2, disabling compactions & flushes 2023-07-16 09:05:44,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:44,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:44,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. after waiting 0 ms 2023-07-16 09:05:44,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:44,827 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-16 09:05:44,827 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-16 09:05:44,827 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:44,828 DEBUG [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1504): Waiting on 1588230740, 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:44,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:44,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7ba9ad3beeb428e3628842526c2eadb2: 2023-07-16 09:05:44,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:44,828 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-16 09:05:44,829 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-16 09:05:44,829 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-16 09:05:44,841 DEBUG [Listener at localhost/37403] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41705,1689498341358 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41705,1689498341358 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-16 09:05:44,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41705] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41705,1689498341358 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-16 09:05:44,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41705] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:49640 deadline: 1689498404843, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41705,1689498341358 aborting 2023-07-16 09:05:44,948 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41705] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41705,1689498341358 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-16 09:05:44,948 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41705] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:49640 deadline: 1689498404948, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41705,1689498341358 aborting 2023-07-16 09:05:45,028 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,41705,1689498341358; all regions closed. 2023-07-16 09:05:45,042 DEBUG [RS:0;jenkins-hbase4:41705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:05:45,043 INFO [RS:0;jenkins-hbase4:41705] regionserver.LeaseManager(133): Closed leases 2023-07-16 09:05:45,043 INFO [RS:0;jenkins-hbase4:41705] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-16 09:05:45,043 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-16 09:05:45,044 INFO [RS:0;jenkins-hbase4:41705] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41705 2023-07-16 09:05:45,052 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:41705 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41705 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-16 09:05:45,053 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:41705 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41705 2023-07-16 09:05:45,053 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:45,053 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:45,053 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:05:45,053 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:05:45,053 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:05:45,056 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41705,1689498341358] 2023-07-16 09:05:45,056 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41705,1689498341358; numProcessing=1 2023-07-16 09:05:45,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:45,056 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,41705,1689498341358 znode expired, triggering replicatorRemoved event 2023-07-16 09:05:45,057 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41705,1689498341358 already deleted, retry=false 2023-07-16 09:05:45,057 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,41705,1689498341358 on jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:05:45,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:45,065 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,41705,1689498341358, splitWal=true, meta=true 2023-07-16 09:05:45,065 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,41705,1689498341358 (carryingMeta=true) jenkins-hbase4.apache.org,41705,1689498341358/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@7e30e1ab[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-16 09:05:45,068 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41705,1689498341358, splitWal=true, meta=true 2023-07-16 09:05:45,069 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41705,1689498341358, splitWal=true, meta=true, isMeta: true 2023-07-16 09:05:45,072 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting 2023-07-16 09:05:45,073 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,41705,1689498341358 WAL count=1, meta=true 2023-07-16 09:05:45,078 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta}] 2023-07-16 09:05:45,088 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:45,089 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta, worker=jenkins-hbase4.apache.org,43645,1689498341402}] 2023-07-16 09:05:45,156 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-16 09:05:45,156 INFO [RS:0;jenkins-hbase4:41705] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,41705,1689498341358; zookeeper connection closed. 2023-07-16 09:05:45,156 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:41705-0x1016d614b290001, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-16 09:05:45,157 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2f27e57] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2f27e57 2023-07-16 09:05:45,252 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:45,253 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-16 09:05:45,256 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55612, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-16 09:05:45,257 DEBUG [hconnection-0x4953d7ba-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41705 this server is in the failed servers list 2023-07-16 09:05:45,257 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43645] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-16 09:05:45,317 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta, size=3.1 K (3185bytes) 2023-07-16 09:05:45,318 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta 2023-07-16 09:05:45,323 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta after 5ms 2023-07-16 09:05:45,327 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta took 11ms 2023-07-16 09:05:45,343 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-16 09:05:45,345 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta so closing down 2023-07-16 09:05:45,345 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-16 09:05:45,350 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-16 09:05:45,352 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta.temp 2023-07-16 09:05:45,353 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-16 09:05:45,365 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-16 09:05:45,367 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta.temp to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-16 09:05:45,367 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-16 09:05:45,368 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta, journal: Splitting hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta, size=3.1 K (3185bytes) at 1689498345317Finishing writing output for hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta so closing down at 1689498345345 (+28 ms)Creating recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta.temp at 1689498345352 (+7 ms)3 split writer threads finished at 1689498345353 (+1 ms)Closed recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689498345365 (+12 ms)Rename recovered edits hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta.temp to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689498345367 (+2 ms)Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689498345368 (+1 ms) 2023-07-16 09:05:45,368 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta 2023-07-16 09:05:45,370 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-16 09:05:45,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.HMaster(3641): Remote procedure done, pid=14 2023-07-16 09:05:45,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] wal.WALSplitUtil(141): Moved hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs 2023-07-16 09:05:45,382 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-16 09:05:45,382 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta, worker=jenkins-hbase4.apache.org,43645,1689498341402 in 290 msec 2023-07-16 09:05:45,384 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:45,387 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-16 09:05:45,387 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C41705%2C1689498341358.meta.1689498342933.meta, worker=jenkins-hbase4.apache.org,43645,1689498341402 in 309 msec 2023-07-16 09:05:45,389 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting dir is empty, no logs to split. 2023-07-16 09:05:45,390 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,41705,1689498341358 WAL count=0, meta=true 2023-07-16 09:05:45,390 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,41705,1689498341358 WAL splitting is done? wals=0, meta=true 2023-07-16 09:05:45,391 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-16 09:05:45,392 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-16 09:05:45,393 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-16 09:05:45,543 DEBUG [jenkins-hbase4:46859] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-16 09:05:45,543 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43645,1689498341402, state=OPENING 2023-07-16 09:05:45,546 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-16 09:05:45,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43645,1689498341402}] 2023-07-16 09:05:45,546 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-16 09:05:45,558 WARN [ReadOnlyZKClient-127.0.0.1:53764@0x3e82be35] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-16 09:05:45,558 DEBUG [hconnection-0x4953d7ba-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:05:45,562 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55624, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:05:45,562 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43645] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:55624 deadline: 1689498405562, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:45,593 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-16 09:05:45,705 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-16 09:05:45,705 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-16 09:05:45,707 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta, suffix=.meta, logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402, archiveDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs, maxLogs=32 2023-07-16 09:05:45,726 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK] 2023-07-16 09:05:45,726 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK] 2023-07-16 09:05:45,730 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta 2023-07-16 09:05:45,730 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK], DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK]] 2023-07-16 09:05:45,730 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:05:45,731 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-16 09:05:45,731 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-16 09:05:45,731 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-16 09:05:45,731 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-16 09:05:45,731 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:45,731 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-16 09:05:45,732 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-16 09:05:45,733 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-16 09:05:45,735 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info 2023-07-16 09:05:45,735 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info 2023-07-16 09:05:45,735 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-16 09:05:45,736 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:45,736 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-16 09:05:45,738 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/rep_barrier 2023-07-16 09:05:45,738 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/rep_barrier 2023-07-16 09:05:45,738 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-16 09:05:45,739 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:45,739 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-16 09:05:45,740 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table 2023-07-16 09:05:45,740 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table 2023-07-16 09:05:45,741 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-16 09:05:45,741 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:45,743 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740 2023-07-16 09:05:45,746 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740 2023-07-16 09:05:45,748 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-16 09:05:45,755 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-16 09:05:45,756 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-16 09:05:45,809 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/.tmp/info/08df6d67a33f4dad83c86275dc3966c0 2023-07-16 09:05:45,862 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/.tmp/table/3539bd4c05b64cff81a8526fe267ef94 2023-07-16 09:05:45,871 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/.tmp/info/08df6d67a33f4dad83c86275dc3966c0 as hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info/08df6d67a33f4dad83c86275dc3966c0 2023-07-16 09:05:45,879 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info/08df6d67a33f4dad83c86275dc3966c0, entries=20, sequenceid=13, filesize=6.9 K 2023-07-16 09:05:45,881 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/.tmp/table/3539bd4c05b64cff81a8526fe267ef94 as hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table/3539bd4c05b64cff81a8526fe267ef94 2023-07-16 09:05:45,889 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table/3539bd4c05b64cff81a8526fe267ef94, entries=4, sequenceid=13, filesize=4.7 K 2023-07-16 09:05:45,889 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 133ms, sequenceid=13, compaction requested=false; wal=null 2023-07-16 09:05:45,891 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-16 09:05:45,893 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-16 09:05:45,896 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-16 09:05:45,898 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-16 09:05:45,901 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-16 09:05:45,902 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61970996, jitterRate=-0.07656019926071167}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-16 09:05:45,902 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-16 09:05:45,904 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689498345699 2023-07-16 09:05:45,906 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-16 09:05:45,907 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-16 09:05:45,907 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43645,1689498341402, state=OPEN 2023-07-16 09:05:45,909 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-16 09:05:45,909 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-16 09:05:45,911 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-16 09:05:45,911 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43645,1689498341402 in 363 msec 2023-07-16 09:05:45,914 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-16 09:05:45,914 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 520 msec 2023-07-16 09:05:45,914 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,41705,1689498341358 had 3 regions 2023-07-16 09:05:45,915 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41705,1689498341358, splitWal=true, meta=true, isMeta: false 2023-07-16 09:05:45,917 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,41705,1689498341358 WAL count=1, meta=false 2023-07-16 09:05:45,918 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754}] 2023-07-16 09:05:45,919 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:45,920 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754, worker=jenkins-hbase4.apache.org,43645,1689498341402}] 2023-07-16 09:05:46,073 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43645] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-16 09:05:46,086 DEBUG [hconnection-0x4953d7ba-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41705 this server is in the failed servers list 2023-07-16 09:05:46,088 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754, size=1.6 K (1665bytes) 2023-07-16 09:05:46,088 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754 2023-07-16 09:05:46,089 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754 after 1ms 2023-07-16 09:05:46,092 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754 took 4ms 2023-07-16 09:05:46,096 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 7ba9ad3beeb428e3628842526c2eadb2: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-16 09:05:46,098 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for f14b930b48c75cc2e0ecf1557d0ca60d: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-16 09:05:46,099 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754 so closing down 2023-07-16 09:05:46,099 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-16 09:05:46,101 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp 2023-07-16 09:05:46,103 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp 2023-07-16 09:05:46,103 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-16 09:05:46,119 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-16 09:05:46,120 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-16 09:05:46,121 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000013 2023-07-16 09:05:46,122 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000005 2023-07-16 09:05:46,122 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 29 ms; skipped=2; WAL=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754, size=1.6 K, length=1665, corrupted=false, cancelled=false 2023-07-16 09:05:46,123 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754, journal: Splitting hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754, size=1.6 K (1665bytes) at 1689498346088Finishing writing output for hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754 so closing down at 1689498346099 (+11 ms)Creating recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp at 1689498346101 (+2 ms)Creating recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp at 1689498346103 (+2 ms)3 split writer threads finished at 1689498346103Closed recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689498346119 (+16 ms)Closed recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689498346120 (+1 ms)Rename recovered edits hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000013 at 1689498346121 (+1 ms)Rename recovered edits hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754.temp to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000005 at 1689498346122 (+1 ms)Processed 14 edits across 2 Regions in 29 ms; skipped=2; WAL=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754, size=1.6 K, length=1665, corrupted=false, cancelled=false at 1689498346122 2023-07-16 09:05:46,123 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754 2023-07-16 09:05:46,123 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-16 09:05:46,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.HMaster(3641): Remote procedure done, pid=18 2023-07-16 09:05:46,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] wal.WALSplitUtil(141): Moved hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting/jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754 to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs 2023-07-16 09:05:46,130 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-16 09:05:46,130 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754, worker=jenkins-hbase4.apache.org,43645,1689498341402 in 207 msec 2023-07-16 09:05:46,132 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:46,134 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-16 09:05:46,134 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C41705%2C1689498341358.1689498342754, worker=jenkins-hbase4.apache.org,43645,1689498341402 in 214 msec 2023-07-16 09:05:46,137 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,41705,1689498341358-splitting dir is empty, no logs to split. 2023-07-16 09:05:46,137 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,41705,1689498341358 WAL count=0, meta=false 2023-07-16 09:05:46,137 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,41705,1689498341358 WAL splitting is done? wals=0, meta=false 2023-07-16 09:05:46,139 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN}, {pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN}] 2023-07-16 09:05:46,140 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN 2023-07-16 09:05:46,140 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN 2023-07-16 09:05:46,141 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-16 09:05:46,142 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-16 09:05:46,142 DEBUG [jenkins-hbase4:46859] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-16 09:05:46,142 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=f14b930b48c75cc2e0ecf1557d0ca60d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:46,143 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689498346142"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689498346142"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689498346142"}]},"ts":"1689498346142"} 2023-07-16 09:05:46,143 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=7ba9ad3beeb428e3628842526c2eadb2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:46,143 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689498346143"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689498346143"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689498346143"}]},"ts":"1689498346143"} 2023-07-16 09:05:46,144 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:41705 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41705 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-16 09:05:46,145 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:41705 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41705 2023-07-16 09:05:46,247 DEBUG [PEWorker-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:05:46,250 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:05:46,252 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure f14b930b48c75cc2e0ecf1557d0ca60d, server=jenkins-hbase4.apache.org,43645,1689498341402}] 2023-07-16 09:05:46,253 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE; OpenRegionProcedure 7ba9ad3beeb428e3628842526c2eadb2, server=jenkins-hbase4.apache.org,43645,1689498341402}] 2023-07-16 09:05:46,291 DEBUG [hconnection-0x4953d7ba-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41705 this server is in the failed servers list 2023-07-16 09:05:46,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:46,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f14b930b48c75cc2e0ecf1557d0ca60d, NAME => 't,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:05:46,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:46,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:46,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:46,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:46,413 INFO [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:46,414 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:46,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7ba9ad3beeb428e3628842526c2eadb2, NAME => 'hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:05:46,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:46,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:05:46,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:46,415 DEBUG [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/f 2023-07-16 09:05:46,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:46,415 DEBUG [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/f 2023-07-16 09:05:46,416 INFO [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f14b930b48c75cc2e0ecf1557d0ca60d columnFamilyName f 2023-07-16 09:05:46,416 INFO [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] regionserver.HStore(310): Store=f14b930b48c75cc2e0ecf1557d0ca60d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:46,417 INFO [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:46,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:46,420 DEBUG [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/info 2023-07-16 09:05:46,420 DEBUG [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/info 2023-07-16 09:05:46,421 INFO [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7ba9ad3beeb428e3628842526c2eadb2 columnFamilyName info 2023-07-16 09:05:46,422 INFO [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] regionserver.HStore(310): Store=7ba9ad3beeb428e3628842526c2eadb2/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:05:46,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:46,424 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:46,425 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000013 2023-07-16 09:05:46,427 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:46,428 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000005 2023-07-16 09:05:46,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000013 2023-07-16 09:05:46,431 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f14b930b48c75cc2e0ecf1557d0ca60d 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-16 09:05:46,432 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000005 2023-07-16 09:05:46,433 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7ba9ad3beeb428e3628842526c2eadb2 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-16 09:05:46,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/.tmp/f/8ac8a91610ea4d699c3197e8c9d13195 2023-07-16 09:05:46,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/.tmp/f/8ac8a91610ea4d699c3197e8c9d13195 as hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/f/8ac8a91610ea4d699c3197e8c9d13195 2023-07-16 09:05:46,465 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/.tmp/info/ab61efb9bb1f4c6fb2b62ca59c3140a0 2023-07-16 09:05:46,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/f/8ac8a91610ea4d699c3197e8c9d13195, entries=10, sequenceid=13, filesize=4.8 K 2023-07-16 09:05:46,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for f14b930b48c75cc2e0ecf1557d0ca60d in 39ms, sequenceid=13, compaction requested=false; wal=null 2023-07-16 09:05:46,470 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-16 09:05:46,472 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/0000000000000000013 2023-07-16 09:05:46,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:05:46,477 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/.tmp/info/ab61efb9bb1f4c6fb2b62ca59c3140a0 as hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/info/ab61efb9bb1f4c6fb2b62ca59c3140a0 2023-07-16 09:05:46,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-16 09:05:46,481 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f14b930b48c75cc2e0ecf1557d0ca60d; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60466755, jitterRate=-0.09897513687610626}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-16 09:05:46,482 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f14b930b48c75cc2e0ecf1557d0ca60d: 2023-07-16 09:05:46,483 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d., pid=21, masterSystemTime=1689498346405 2023-07-16 09:05:46,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:46,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:46,486 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/info/ab61efb9bb1f4c6fb2b62ca59c3140a0, entries=2, sequenceid=5, filesize=4.8 K 2023-07-16 09:05:46,487 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 7ba9ad3beeb428e3628842526c2eadb2 in 53ms, sequenceid=5, compaction requested=false; wal=null 2023-07-16 09:05:46,487 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=f14b930b48c75cc2e0ecf1557d0ca60d, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:46,487 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-16 09:05:46,487 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689498346486"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689498346486"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689498346486"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689498346486"}]},"ts":"1689498346486"} 2023-07-16 09:05:46,489 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/0000000000000000005 2023-07-16 09:05:46,492 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-16 09:05:46,492 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure f14b930b48c75cc2e0ecf1557d0ca60d, server=jenkins-hbase4.apache.org,43645,1689498341402 in 237 msec 2023-07-16 09:05:46,493 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:05:46,494 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN in 353 msec 2023-07-16 09:05:46,496 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-16 09:05:46,497 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7ba9ad3beeb428e3628842526c2eadb2; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72597432, jitterRate=0.08178603649139404}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-16 09:05:46,497 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7ba9ad3beeb428e3628842526c2eadb2: 2023-07-16 09:05:46,498 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2., pid=22, masterSystemTime=1689498346405 2023-07-16 09:05:46,500 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:46,500 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:05:46,501 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=7ba9ad3beeb428e3628842526c2eadb2, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:46,501 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689498346500"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689498346500"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689498346500"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689498346500"}]},"ts":"1689498346500"} 2023-07-16 09:05:46,506 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-07-16 09:05:46,506 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; OpenRegionProcedure 7ba9ad3beeb428e3628842526c2eadb2, server=jenkins-hbase4.apache.org,43645,1689498341402 in 250 msec 2023-07-16 09:05:46,508 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-07-16 09:05:46,509 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,41705,1689498341358 after splitting done 2023-07-16 09:05:46,509 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN in 367 msec 2023-07-16 09:05:46,509 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase4.apache.org,41705,1689498341358 from processing; numProcessing=0 2023-07-16 09:05:46,512 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,41705,1689498341358, splitWal=true, meta=true in 1.4500 sec 2023-07-16 09:05:46,602 DEBUG [Listener at localhost/37403] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e82be35 to 127.0.0.1:53764 2023-07-16 09:05:46,602 DEBUG [Listener at localhost/37403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:05:46,602 INFO [Listener at localhost/37403] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-16 09:05:46,615 INFO [Listener at localhost/37403] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=271 (was 294), OpenFileDescriptor=580 (was 581), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=216 (was 216), ProcessCount=169 (was 169), AvailableMemoryMB=5800 (was 5862) 2023-07-16 09:05:46,626 INFO [Listener at localhost/37403] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=271, OpenFileDescriptor=580, MaxFileDescriptor=60000, SystemLoadAverage=216, ProcessCount=169, AvailableMemoryMB=5800 2023-07-16 09:05:46,627 INFO [Listener at localhost/37403] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:41705 2023-07-16 09:05:46,639 INFO [Listener at localhost/37403] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-16 09:05:46,639 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:46,639 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:46,639 INFO [Listener at localhost/37403] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-16 09:05:46,639 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-16 09:05:46,639 INFO [Listener at localhost/37403] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-16 09:05:46,639 INFO [Listener at localhost/37403] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-16 09:05:46,640 INFO [Listener at localhost/37403] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36585 2023-07-16 09:05:46,641 INFO [Listener at localhost/37403] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-16 09:05:46,642 DEBUG [Listener at localhost/37403] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-16 09:05:46,643 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:46,645 INFO [Listener at localhost/37403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-16 09:05:46,646 INFO [Listener at localhost/37403] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36585 connecting to ZooKeeper ensemble=127.0.0.1:53764 2023-07-16 09:05:46,650 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:365850x0, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-16 09:05:46,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36585-0x1016d614b290008 connected 2023-07-16 09:05:46,652 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(162): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-16 09:05:46,653 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(162): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-16 09:05:46,654 DEBUG [Listener at localhost/37403] zookeeper.ZKUtil(164): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-16 09:05:46,654 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36585 2023-07-16 09:05:46,654 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36585 2023-07-16 09:05:46,655 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36585 2023-07-16 09:05:46,655 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36585 2023-07-16 09:05:46,655 DEBUG [Listener at localhost/37403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36585 2023-07-16 09:05:46,657 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(951): ClusterId : 1975af79-dd1e-4db1-8abc-56f2cdde95d7 2023-07-16 09:05:46,657 DEBUG [RS:2;jenkins-hbase4:36585] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-16 09:05:46,661 DEBUG [RS:2;jenkins-hbase4:36585] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-16 09:05:46,661 DEBUG [RS:2;jenkins-hbase4:36585] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-16 09:05:46,663 DEBUG [RS:2;jenkins-hbase4:36585] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-16 09:05:46,664 DEBUG [RS:2;jenkins-hbase4:36585] zookeeper.ReadOnlyZKClient(139): Connect 0x0c4f7e07 to 127.0.0.1:53764 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-16 09:05:46,667 DEBUG [RS:2;jenkins-hbase4:36585] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28891fe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-16 09:05:46,667 DEBUG [RS:2;jenkins-hbase4:36585] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b71577, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-16 09:05:46,676 DEBUG [RS:2;jenkins-hbase4:36585] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:36585 2023-07-16 09:05:46,676 INFO [RS:2;jenkins-hbase4:36585] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-16 09:05:46,676 INFO [RS:2;jenkins-hbase4:36585] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-16 09:05:46,676 DEBUG [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1022): About to register with Master. 2023-07-16 09:05:46,677 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,46859,1689498340206 with isa=jenkins-hbase4.apache.org/172.31.14.131:36585, startcode=1689498346638 2023-07-16 09:05:46,677 DEBUG [RS:2;jenkins-hbase4:36585] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-16 09:05:46,679 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57811, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-16 09:05:46,680 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46859] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:05:46,680 DEBUG [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76 2023-07-16 09:05:46,680 DEBUG [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:36827 2023-07-16 09:05:46,680 DEBUG [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-16 09:05:46,682 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:05:46,682 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:05:46,682 DEBUG [RS:2;jenkins-hbase4:36585] zookeeper.ZKUtil(162): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:05:46,682 WARN [RS:2;jenkins-hbase4:36585] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-16 09:05:46,682 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36585,1689498346638] 2023-07-16 09:05:46,682 INFO [RS:2;jenkins-hbase4:36585] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-16 09:05:46,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:05:46,683 DEBUG [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1948): logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:05:46,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:46,687 DEBUG [RS:2;jenkins-hbase4:36585] zookeeper.ZKUtil(162): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:05:46,687 DEBUG [RS:2;jenkins-hbase4:36585] zookeeper.ZKUtil(162): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:05:46,688 DEBUG [RS:2;jenkins-hbase4:36585] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-16 09:05:46,688 INFO [RS:2;jenkins-hbase4:36585] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-16 09:05:46,690 INFO [RS:2;jenkins-hbase4:36585] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-16 09:05:46,690 INFO [RS:2;jenkins-hbase4:36585] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-16 09:05:46,690 INFO [RS:2;jenkins-hbase4:36585] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:46,690 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-16 09:05:46,692 INFO [RS:2;jenkins-hbase4:36585] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:46,692 DEBUG [RS:2;jenkins-hbase4:36585] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-16 09:05:46,695 INFO [RS:2;jenkins-hbase4:36585] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:46,695 INFO [RS:2;jenkins-hbase4:36585] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:46,695 INFO [RS:2;jenkins-hbase4:36585] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:46,706 INFO [RS:2;jenkins-hbase4:36585] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-16 09:05:46,706 INFO [RS:2;jenkins-hbase4:36585] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36585,1689498346638-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-16 09:05:46,717 INFO [RS:2;jenkins-hbase4:36585] regionserver.Replication(203): jenkins-hbase4.apache.org,36585,1689498346638 started 2023-07-16 09:05:46,717 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,36585,1689498346638, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36585, sessionid=0x1016d614b290008 2023-07-16 09:05:46,717 INFO [Listener at localhost/37403] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:36585,5,FailOnTimeoutGroup] 2023-07-16 09:05:46,717 DEBUG [RS:2;jenkins-hbase4:36585] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-16 09:05:46,717 DEBUG [RS:2;jenkins-hbase4:36585] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:05:46,717 INFO [Listener at localhost/37403] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:05:46,717 DEBUG [RS:2;jenkins-hbase4:36585] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36585,1689498346638' 2023-07-16 09:05:46,717 DEBUG [RS:2;jenkins-hbase4:36585] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-16 09:05:46,717 INFO [Listener at localhost/37403] client.TestScannerTimeout(177): START ************ test3686b 2023-07-16 09:05:46,718 DEBUG [RS:2;jenkins-hbase4:36585] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-16 09:05:46,718 DEBUG [RS:2;jenkins-hbase4:36585] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-16 09:05:46,718 DEBUG [RS:2;jenkins-hbase4:36585] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-16 09:05:46,718 DEBUG [RS:2;jenkins-hbase4:36585] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:05:46,718 DEBUG [RS:2;jenkins-hbase4:36585] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36585,1689498346638' 2023-07-16 09:05:46,718 DEBUG [RS:2;jenkins-hbase4:36585] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-16 09:05:46,718 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:41705 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41705 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-16 09:05:46,719 DEBUG [RS:2;jenkins-hbase4:36585] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-16 09:05:46,719 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:41705 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41705 2023-07-16 09:05:46,720 DEBUG [RS:2;jenkins-hbase4:36585] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-16 09:05:46,720 INFO [RS:2;jenkins-hbase4:36585] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-16 09:05:46,720 INFO [RS:2;jenkins-hbase4:36585] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-16 09:05:46,822 DEBUG [hconnection-0x7bfc2c0d-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:05:46,823 INFO [RS:2;jenkins-hbase4:36585] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36585%2C1689498346638, suffix=, logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,36585,1689498346638, archiveDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs, maxLogs=32 2023-07-16 09:05:46,825 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:05:46,829 DEBUG [Listener at localhost/37403] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-16 09:05:46,829 DEBUG [Listener at localhost/37403] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:05:46,830 DEBUG [Listener at localhost/37403] zookeeper.ReadOnlyZKClient(139): Connect 0x034dd6e0 to 127.0.0.1:53764 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-16 09:05:46,835 DEBUG [Listener at localhost/37403] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45d733dc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-16 09:05:46,840 DEBUG [hconnection-0x2476c7e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:05:46,844 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:05:46,861 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK] 2023-07-16 09:05:46,867 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK] 2023-07-16 09:05:46,869 INFO [RS:2;jenkins-hbase4:36585] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,36585,1689498346638/jenkins-hbase4.apache.org%2C36585%2C1689498346638.1689498346826 2023-07-16 09:05:46,870 DEBUG [RS:2;jenkins-hbase4:36585] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK], DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK]] 2023-07-16 09:05:48,542 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-16 09:05:50,368 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-16 09:05:50,369 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-16 09:06:01,856 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 60600502487875591 expired clientIPAndPort=172.31.14.131:55644, userName=jenkins, regionInfo=t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:03,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43645] regionserver.RSRpcServices(3094): Client tried to access missing scanner 60600502487875591 2023-07-16 09:06:03,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43645] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 41 connection: 172.31.14.131:55644 deadline: 1689498423862, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '60600502487875591'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-16 09:06:03,864 DEBUG [Listener at localhost/37403] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '60600502487875591'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '60600502487875591'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '60600502487875591'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-16 09:06:03,871 DEBUG [Listener at localhost/37403] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x034dd6e0 to 127.0.0.1:53764 2023-07-16 09:06:03,871 DEBUG [Listener at localhost/37403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:06:03,871 INFO [Listener at localhost/37403] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-16 09:06:03,890 INFO [Listener at localhost/37403] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=297 (was 271) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76-prefix:jenkins-hbase4.apache.org,36585,1689498346638 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53764@0x0c4f7e07 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1667281059.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53764@0x034dd6e0-SendThread(127.0.0.1:53764) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_629323691_17 at /127.0.0.1:35774 [Receiving block BP-887411987-172.31.14.131-1689498337264:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53764@0x034dd6e0-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: jenkins-hbase4:36585Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-887411987-172.31.14.131-1689498337264:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-887411987-172.31.14.131-1689498337264:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53764@0x0c4f7e07-SendThread(127.0.0.1:53764) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS:2;jenkins-hbase4:36585-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:36827 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53764@0x0c4f7e07-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:53764@0x034dd6e0 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1667281059.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36585 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_629323691_17 at /127.0.0.1:55444 [Receiving block BP-887411987-172.31.14.131-1689498337264:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:36585 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=574 (was 580), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=246 (was 216) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6262 (was 5800) - AvailableMemoryMB LEAK? - 2023-07-16 09:06:03,904 INFO [Listener at localhost/37403] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=297, OpenFileDescriptor=574, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=169, AvailableMemoryMB=6261 2023-07-16 09:06:03,904 INFO [Listener at localhost/37403] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:41705 2023-07-16 09:06:03,904 INFO [Listener at localhost/37403] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,41705,1689498341358 2023-07-16 09:06:03,905 INFO [Listener at localhost/37403] client.TestScannerTimeout(104): START************ test2772 2023-07-16 09:06:03,908 DEBUG [Listener at localhost/37403] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-16 09:06:03,908 DEBUG [Listener at localhost/37403] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:03,908 DEBUG [Listener at localhost/37403] zookeeper.ReadOnlyZKClient(139): Connect 0x52db3f80 to 127.0.0.1:53764 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-16 09:06:03,913 DEBUG [Listener at localhost/37403] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49312bf5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-16 09:06:03,913 ERROR [Listener at localhost/37403] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,43645,1689498341402: die! ***** 2023-07-16 09:06:03,913 ERROR [Listener at localhost/37403] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-16 09:06:03,914 DEBUG [Listener at localhost/37403] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-16 09:06:03,914 DEBUG [Listener at localhost/37403] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-16 09:06:03,921 DEBUG [Listener at localhost/37403] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-16 09:06:03,923 DEBUG [Listener at localhost/37403] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-16 09:06:03,943 INFO [Listener at localhost/37403] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 985661440, "init": 513802240, "max": 2051014656, "used": 274416272 }, "NonHeapMemoryUsage": { "committed": 124542976, "init": 2555904, "max": -1, "used": 122081000 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13256, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 63, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 63, "QueueCallTime_min": 0, "QueueCallTime_max": 1, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 1, "QueueCallTime_75th_percentile": 1, "QueueCallTime_90th_percentile": 1, "QueueCallTime_95th_percentile": 1, "QueueCallTime_98th_percentile": 1, "QueueCallTime_99th_percentile": 1, "QueueCallTime_99.9th_percentile": 1, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 63, "TotalCallTime_min": 1, "TotalCallTime_max": 2, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 2, "TotalCallTime_95th_percentile": 2, "TotalCallTime_98th_percentile": 2, "TotalCallTime_99th_percentile": 2, "TotalCallTime_99.9th_percentile": 2, "TotalCallTime_TimeRangeCount_0-1": 1, "TotalCallTime_TimeRangeCount_1-3": 1, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 63, "ResponseSize_min": 195, "ResponseSize_max": 677, "ResponseSize_mean": 436, "ResponseSize_25th_percentile": 195, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 676, "ResponseSize_95th_percentile": 676, "ResponseSize_98th_percentile": 676, "ResponseSize_99th_percentile": 676, "ResponseSize_99.9th_percentile": 676, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 5, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 63, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16922 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:53764", "tag.serverName": "jenkins-hbase4.apache.org,43645,1689498341402", "tag.clusterId": "1975af79-dd1e-4db1-8abc-56f2cdde95d7", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 4, "maxStoreFileCount": 1, "memStoreSize": 1792, "storeFileSize": 21711, "maxStoreFileAge": 16749, "minStoreFileAge": 16095, "avgStoreFileAge": 16412, "numReferenceFiles": 0, "regionServerStartTime": 1689498341402, "averageRegionSize": 7834, "storeFileIndexSize": 1368, "staticIndexSize": 221, "staticBloomSize": 4, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 2492, "localBytesRead": 2492, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819799248, "blockCacheCount": 3, "blockCacheDataBlockCount": 3, "blockCacheSize": 606640, "blockCacheCountHitPercent": 85.0, "blockCacheExpressHitPercent": 85.0, "l1CacheSize": 606640, "l1CacheFreeSize": 819799248, "l1CacheCount": 3, "l1CacheEvictionCount": 0, "l1CacheHitCount": 17, "l1CacheMissCount": 3, "l1CacheHitRatio": 0.85, "l1CacheMissRatio": 0.15, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 8750, "ByteBuffAllocatorPoolAllocationBytes": 798720, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 18, "totalRowActionRequestCount": 22, "readRequestCount": 18, "filteredReadRequestCount": 0, "writeRequestCount": 4, "rpcGetRequestCount": 4, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 10, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 4, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 17, "blockCacheHitCountPrimary": 17, "blockCacheHitCachingCount": 17, "blockCacheMissCount": 3, "blockCacheMissCountPrimary": 3, "blockCacheMissCachingCount": 3, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 3, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 17, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 36, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 2742, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 9, "ScanTime_min": 1, "ScanTime_max": 1, "ScanTime_mean": 1, "ScanTime_25th_percentile": 1, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 4, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 3, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 3, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 21711, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 3, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 9, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 155, "ScanSize_median": 156, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 2742, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 3, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 52, "ServerReadQueryPerSecond_mean_rate": 2.4338070056314556, "ServerReadQueryPerSecond_1min_rate": 7.164967204256925, "ServerReadQueryPerSecond_5min_rate": 8.751310705406569, "ServerReadQueryPerSecond_15min_rate": 9.047937375158883, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.216903477128605, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-16 09:06:03,944 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46859] master.MasterRpcServices(612): jenkins-hbase4.apache.org,43645,1689498341402 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,43645,1689498341402: die! ***** 2023-07-16 09:06:03,946 INFO [Listener at localhost/37403] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,43645,1689498341402' ***** 2023-07-16 09:06:03,947 INFO [Listener at localhost/37403] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-16 09:06:03,947 INFO [RS:1;jenkins-hbase4:43645] regionserver.HeapMemoryManager(220): Stopping 2023-07-16 09:06:03,948 INFO [RS:1;jenkins-hbase4:43645] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-16 09:06:03,948 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-16 09:06:03,948 INFO [RS:1;jenkins-hbase4:43645] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-16 09:06:03,948 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(3305): Received CLOSE for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:03,954 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(3305): Received CLOSE for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:06:03,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f14b930b48c75cc2e0ecf1557d0ca60d, disabling compactions & flushes 2023-07-16 09:06:03,955 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:06:03,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:03,955 DEBUG [RS:1;jenkins-hbase4:43645] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x11caf6b1 to 127.0.0.1:53764 2023-07-16 09:06:03,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:03,956 DEBUG [RS:1;jenkins-hbase4:43645] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:06:03,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. after waiting 0 ms 2023-07-16 09:06:03,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:03,956 INFO [RS:1;jenkins-hbase4:43645] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-16 09:06:03,956 INFO [RS:1;jenkins-hbase4:43645] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-16 09:06:03,956 INFO [RS:1;jenkins-hbase4:43645] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-16 09:06:03,956 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-16 09:06:03,959 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-16 09:06:03,959 DEBUG [hconnection-0x488f3bae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:06:03,959 DEBUG [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, f14b930b48c75cc2e0ecf1557d0ca60d=t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d., 7ba9ad3beeb428e3628842526c2eadb2=hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.} 2023-07-16 09:06:03,959 DEBUG [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1504): Waiting on 1588230740, 7ba9ad3beeb428e3628842526c2eadb2, f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:03,961 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-16 09:06:03,962 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-16 09:06:03,962 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-16 09:06:03,962 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-16 09:06:03,962 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-16 09:06:03,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:03,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f14b930b48c75cc2e0ecf1557d0ca60d: 2023-07-16 09:06:03,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:03,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7ba9ad3beeb428e3628842526c2eadb2, disabling compactions & flushes 2023-07-16 09:06:03,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:03,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:03,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. after waiting 0 ms 2023-07-16 09:06:03,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:03,969 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34710, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:06:03,971 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43645] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43645,1689498341402 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-16 09:06:03,971 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43645] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:34710 deadline: 1689498423970, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43645,1689498341402 aborting 2023-07-16 09:06:03,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:03,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7ba9ad3beeb428e3628842526c2eadb2: 2023-07-16 09:06:03,976 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 1792 in region hbase:meta,,1.1588230740 2023-07-16 09:06:03,976 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-16 09:06:03,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:03,977 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-16 09:06:03,977 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-16 09:06:03,977 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-16 09:06:04,077 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43645] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43645,1689498341402 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-16 09:06:04,077 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43645] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:34710 deadline: 1689498424076, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43645,1689498341402 aborting 2023-07-16 09:06:04,159 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,43645,1689498341402; all regions closed. 2023-07-16 09:06:04,176 DEBUG [RS:1;jenkins-hbase4:43645] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:06:04,176 INFO [RS:1;jenkins-hbase4:43645] regionserver.LeaseManager(133): Closed leases 2023-07-16 09:06:04,176 INFO [RS:1;jenkins-hbase4:43645] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-16 09:06:04,177 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-16 09:06:04,177 INFO [RS:1;jenkins-hbase4:43645] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43645 2023-07-16 09:06:04,181 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:06:04,181 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:06:04,181 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 2023-07-16 09:06:04,181 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:06:04,182 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:06:04,182 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43645,1689498341402] 2023-07-16 09:06:04,182 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43645,1689498341402; numProcessing=1 2023-07-16 09:06:04,184 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43645,1689498341402 already deleted, retry=false 2023-07-16 09:06:04,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:04,184 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,43645,1689498341402 on jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:06:04,185 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,43645,1689498341402 znode expired, triggering replicatorRemoved event 2023-07-16 09:06:04,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:04,186 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=23, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,43645,1689498341402, splitWal=true, meta=true 2023-07-16 09:06:04,186 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=23 for jenkins-hbase4.apache.org,43645,1689498341402 (carryingMeta=true) jenkins-hbase4.apache.org,43645,1689498341402/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@51a7752[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-16 09:06:04,188 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=23, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43645,1689498341402, splitWal=true, meta=true 2023-07-16 09:06:04,189 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43645,1689498341402, splitWal=true, meta=true, isMeta: true 2023-07-16 09:06:04,193 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting 2023-07-16 09:06:04,195 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,43645,1689498341402 WAL count=1, meta=true 2023-07-16 09:06:04,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta}] 2023-07-16 09:06:04,197 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:04,198 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta, worker=jenkins-hbase4.apache.org,36585,1689498346638}] 2023-07-16 09:06:04,206 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-16 09:06:04,281 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43645 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43645 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-16 09:06:04,282 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43645 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43645 2023-07-16 09:06:04,284 INFO [RS:1;jenkins-hbase4:43645] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,43645,1689498341402; zookeeper connection closed. 2023-07-16 09:06:04,284 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-16 09:06:04,284 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:43645-0x1016d614b290002, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-16 09:06:04,284 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@16fe154b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@16fe154b 2023-07-16 09:06:04,351 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:04,351 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-16 09:06:04,353 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-16 09:06:04,353 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36585] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-16 09:06:04,370 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta, size=2.3 K (2403bytes) 2023-07-16 09:06:04,370 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta 2023-07-16 09:06:04,370 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta after 0ms 2023-07-16 09:06:04,376 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta took 7ms 2023-07-16 09:06:04,381 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-16 09:06:04,382 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta so closing down 2023-07-16 09:06:04,382 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-16 09:06:04,388 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta.temp 2023-07-16 09:06:04,388 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-16 09:06:04,400 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) 2023-07-16 09:06:04,402 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta.temp to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-16 09:06:04,402 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 5 edits across 1 Regions in 25 ms; skipped=1; WAL=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false 2023-07-16 09:06:04,402 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta, journal: Splitting hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta, size=2.3 K (2403bytes) at 1689498364370Finishing writing output for hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta so closing down at 1689498364382 (+12 ms)Creating recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta.temp at 1689498364388 (+6 ms)3 split writer threads finished at 1689498364388Closed recovered edits writer path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) at 1689498364400 (+12 ms)Rename recovered edits hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta.temp to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 at 1689498364402 (+2 ms)Processed 5 edits across 1 Regions in 25 ms; skipped=1; WAL=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false at 1689498364402 2023-07-16 09:06:04,402 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta 2023-07-16 09:06:04,402 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-16 09:06:04,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.HMaster(3641): Remote procedure done, pid=25 2023-07-16 09:06:04,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] wal.WALSplitUtil(141): Moved hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs 2023-07-16 09:06:04,410 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-16 09:06:04,410 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta, worker=jenkins-hbase4.apache.org,36585,1689498346638 in 209 msec 2023-07-16 09:06:04,411 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:04,413 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=23 2023-07-16 09:06:04,413 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43645%2C1689498341402.meta.1689498345709.meta, worker=jenkins-hbase4.apache.org,36585,1689498346638 in 216 msec 2023-07-16 09:06:04,416 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting dir is empty, no logs to split. 2023-07-16 09:06:04,416 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,43645,1689498341402 WAL count=0, meta=true 2023-07-16 09:06:04,416 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43645,1689498341402 WAL splitting is done? wals=0, meta=true 2023-07-16 09:06:04,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-16 09:06:04,418 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-16 09:06:04,420 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-16 09:06:04,570 DEBUG [jenkins-hbase4:46859] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-16 09:06:04,570 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36585,1689498346638, state=OPENING 2023-07-16 09:06:04,577 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-16 09:06:04,577 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36585,1689498346638}] 2023-07-16 09:06:04,577 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-16 09:06:04,584 WARN [ReadOnlyZKClient-127.0.0.1:53764@0x52db3f80] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-16 09:06:04,584 DEBUG [hconnection-0x488f3bae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:06:04,588 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:06:04,588 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36585] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:48120 deadline: 1689498424588, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:04,735 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-16 09:06:04,735 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-16 09:06:04,737 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36585%2C1689498346638.meta, suffix=.meta, logDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,36585,1689498346638, archiveDir=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs, maxLogs=32 2023-07-16 09:06:04,755 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK] 2023-07-16 09:06:04,755 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK] 2023-07-16 09:06:04,757 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,36585,1689498346638/jenkins-hbase4.apache.org%2C36585%2C1689498346638.meta.1689498364738.meta 2023-07-16 09:06:04,757 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42717,DS-1f9749cf-f995-46a7-b446-fc48533aaccf,DISK], DatanodeInfoWithStorage[127.0.0.1:41565,DS-9dc60255-3c2c-40eb-865f-39ad3276dd9a,DISK]] 2023-07-16 09:06:04,757 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:06:04,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-16 09:06:04,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-16 09:06:04,758 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-16 09:06:04,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-16 09:06:04,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:06:04,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-16 09:06:04,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-16 09:06:04,760 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-16 09:06:04,761 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info 2023-07-16 09:06:04,761 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info 2023-07-16 09:06:04,762 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-16 09:06:04,774 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info/08df6d67a33f4dad83c86275dc3966c0 2023-07-16 09:06:04,774 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:06:04,775 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-16 09:06:04,776 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/rep_barrier 2023-07-16 09:06:04,776 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/rep_barrier 2023-07-16 09:06:04,777 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-16 09:06:04,778 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:06:04,778 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-16 09:06:04,779 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table 2023-07-16 09:06:04,779 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table 2023-07-16 09:06:04,779 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-16 09:06:04,791 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/table/3539bd4c05b64cff81a8526fe267ef94 2023-07-16 09:06:04,791 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:06:04,792 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740 2023-07-16 09:06:04,794 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740 2023-07-16 09:06:04,795 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-16 09:06:04,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 16, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=19, path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-16 09:06:04,799 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-16 09:06:04,817 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=19 (bloomFilter=false), to=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/.tmp/info/dbf6266302c648a38ff4e0d114934b33 2023-07-16 09:06:04,825 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/.tmp/info/dbf6266302c648a38ff4e0d114934b33 as hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info/dbf6266302c648a38ff4e0d114934b33 2023-07-16 09:06:04,833 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info/dbf6266302c648a38ff4e0d114934b33, entries=16, sequenceid=19, filesize=6.4 K 2023-07-16 09:06:04,833 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 34ms, sequenceid=19, compaction requested=false; wal=null 2023-07-16 09:06:04,834 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-16 09:06:04,837 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-16 09:06:04,839 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-16 09:06:04,843 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=13 2023-07-16 09:06:04,844 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=20; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66863880, jitterRate=-0.0036505460739135742}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-16 09:06:04,844 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-16 09:06:04,845 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=27, masterSystemTime=1689498364730 2023-07-16 09:06:04,848 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-16 09:06:04,849 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-16 09:06:04,850 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36585,1689498346638, state=OPEN 2023-07-16 09:06:04,851 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-16 09:06:04,851 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-16 09:06:04,854 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-16 09:06:04,854 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36585,1689498346638 in 274 msec 2023-07-16 09:06:04,856 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=23 2023-07-16 09:06:04,856 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,43645,1689498341402 had 3 regions 2023-07-16 09:06:04,856 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 437 msec 2023-07-16 09:06:04,857 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43645,1689498341402, splitWal=true, meta=true, isMeta: false 2023-07-16 09:06:04,859 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,43645,1689498341402 WAL count=1, meta=false 2023-07-16 09:06:04,859 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754}] 2023-07-16 09:06:04,861 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:04,862 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754, worker=jenkins-hbase4.apache.org,36585,1689498346638}] 2023-07-16 09:06:05,015 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36585] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=29 2023-07-16 09:06:05,031 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754, size=721 (721bytes) 2023-07-16 09:06:05,031 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754 2023-07-16 09:06:05,032 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754 after 1ms 2023-07-16 09:06:05,036 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754 took 5ms 2023-07-16 09:06:05,045 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for f14b930b48c75cc2e0ecf1557d0ca60d: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-16 09:06:05,047 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 7ba9ad3beeb428e3628842526c2eadb2: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-16 09:06:05,047 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754 so closing down 2023-07-16 09:06:05,047 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-16 09:06:05,048 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-16 09:06:05,048 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 9 ms; skipped=2; WAL=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754, size=721, length=721, corrupted=false, cancelled=false 2023-07-16 09:06:05,048 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754, journal: Splitting hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754, size=721 (721bytes) at 1689498365031Finishing writing output for hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754 so closing down at 1689498365047 (+16 ms)3 split writer threads finished at 1689498365048 (+1 ms)Processed 2 edits across 0 Regions in 9 ms; skipped=2; WAL=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754, size=721, length=721, corrupted=false, cancelled=false at 1689498365048 2023-07-16 09:06:05,048 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754 2023-07-16 09:06:05,048 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=29 2023-07-16 09:06:05,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] master.HMaster(3641): Remote procedure done, pid=29 2023-07-16 09:06:05,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46859] wal.WALSplitUtil(141): Moved hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting/jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754 to hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs 2023-07-16 09:06:05,057 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-07-16 09:06:05,057 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754, worker=jenkins-hbase4.apache.org,36585,1689498346638 in 192 msec 2023-07-16 09:06:05,058 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:05,060 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=23 2023-07-16 09:06:05,061 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43645%2C1689498341402.1689498342754, worker=jenkins-hbase4.apache.org,36585,1689498346638 in 199 msec 2023-07-16 09:06:05,063 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/WALs/jenkins-hbase4.apache.org,43645,1689498341402-splitting dir is empty, no logs to split. 2023-07-16 09:06:05,063 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,43645,1689498341402 WAL count=0, meta=false 2023-07-16 09:06:05,063 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43645,1689498341402 WAL splitting is done? wals=0, meta=false 2023-07-16 09:06:05,065 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN}, {pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN}] 2023-07-16 09:06:05,066 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN 2023-07-16 09:06:05,066 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN 2023-07-16 09:06:05,067 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-16 09:06:05,067 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-16 09:06:05,099 DEBUG [hconnection-0x488f3bae-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43645 this server is in the failed servers list 2023-07-16 09:06:05,207 DEBUG [hconnection-0x488f3bae-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43645 this server is in the failed servers list 2023-07-16 09:06:05,218 DEBUG [jenkins-hbase4:46859] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-16 09:06:05,218 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=f14b930b48c75cc2e0ecf1557d0ca60d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:05,218 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=7ba9ad3beeb428e3628842526c2eadb2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:05,218 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689498365218"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689498365218"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689498365218"}]},"ts":"1689498365218"} 2023-07-16 09:06:05,218 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689498365218"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689498365218"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689498365218"}]},"ts":"1689498365218"} 2023-07-16 09:06:05,219 WARN [RS-EventLoopGroup-4-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43645 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43645 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-16 09:06:05,220 DEBUG [RS-EventLoopGroup-4-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43645 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43645 2023-07-16 09:06:05,321 DEBUG [PEWorker-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-16 09:06:05,323 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-16 09:06:05,325 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=30, state=RUNNABLE; OpenRegionProcedure 7ba9ad3beeb428e3628842526c2eadb2, server=jenkins-hbase4.apache.org,36585,1689498346638}] 2023-07-16 09:06:05,326 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure f14b930b48c75cc2e0ecf1557d0ca60d, server=jenkins-hbase4.apache.org,36585,1689498346638}] 2023-07-16 09:06:05,413 DEBUG [hconnection-0x488f3bae-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43645 this server is in the failed servers list 2023-07-16 09:06:05,482 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:05,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f14b930b48c75cc2e0ecf1557d0ca60d, NAME => 't,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:06:05,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:05,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:06:05,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:05,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:05,485 INFO [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:05,486 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:05,486 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7ba9ad3beeb428e3628842526c2eadb2, NAME => 'hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.', STARTKEY => '', ENDKEY => ''} 2023-07-16 09:06:05,486 DEBUG [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/f 2023-07-16 09:06:05,487 DEBUG [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/f 2023-07-16 09:06:05,487 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:06:05,487 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-16 09:06:05,487 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:06:05,487 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:06:05,487 INFO [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f14b930b48c75cc2e0ecf1557d0ca60d columnFamilyName f 2023-07-16 09:06:05,488 INFO [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:06:05,490 DEBUG [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/info 2023-07-16 09:06:05,490 DEBUG [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/info 2023-07-16 09:06:05,490 INFO [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7ba9ad3beeb428e3628842526c2eadb2 columnFamilyName info 2023-07-16 09:06:05,496 DEBUG [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] regionserver.HStore(539): loaded hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/f/8ac8a91610ea4d699c3197e8c9d13195 2023-07-16 09:06:05,500 INFO [StoreOpener-f14b930b48c75cc2e0ecf1557d0ca60d-1] regionserver.HStore(310): Store=f14b930b48c75cc2e0ecf1557d0ca60d/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:06:05,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:05,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:05,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:05,510 DEBUG [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] regionserver.HStore(539): loaded hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/info/ab61efb9bb1f4c6fb2b62ca59c3140a0 2023-07-16 09:06:05,510 INFO [StoreOpener-7ba9ad3beeb428e3628842526c2eadb2-1] regionserver.HStore(310): Store=7ba9ad3beeb428e3628842526c2eadb2/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-16 09:06:05,511 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:06:05,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-16 09:06:05,513 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:06:05,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f14b930b48c75cc2e0ecf1557d0ca60d; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70931041, jitterRate=0.05695487558841705}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-16 09:06:05,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f14b930b48c75cc2e0ecf1557d0ca60d: 2023-07-16 09:06:05,516 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d., pid=33, masterSystemTime=1689498365478 2023-07-16 09:06:05,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:05,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:05,520 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=f14b930b48c75cc2e0ecf1557d0ca60d, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:05,520 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689498365519"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689498365519"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689498365519"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689498365519"}]},"ts":"1689498365519"} 2023-07-16 09:06:05,525 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-07-16 09:06:05,525 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure f14b930b48c75cc2e0ecf1557d0ca60d, server=jenkins-hbase4.apache.org,36585,1689498346638 in 196 msec 2023-07-16 09:06:05,527 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:06:05,529 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=t, region=f14b930b48c75cc2e0ecf1557d0ca60d, ASSIGN in 460 msec 2023-07-16 09:06:05,531 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-16 09:06:05,532 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7ba9ad3beeb428e3628842526c2eadb2; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63045587, jitterRate=-0.06054754555225372}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-16 09:06:05,532 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7ba9ad3beeb428e3628842526c2eadb2: 2023-07-16 09:06:05,533 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2., pid=32, masterSystemTime=1689498365478 2023-07-16 09:06:05,535 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:05,535 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:05,536 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=7ba9ad3beeb428e3628842526c2eadb2, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:05,536 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689498365536"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689498365536"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689498365536"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689498365536"}]},"ts":"1689498365536"} 2023-07-16 09:06:05,540 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=30 2023-07-16 09:06:05,541 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=30, state=SUCCESS; OpenRegionProcedure 7ba9ad3beeb428e3628842526c2eadb2, server=jenkins-hbase4.apache.org,36585,1689498346638 in 213 msec 2023-07-16 09:06:05,543 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=23 2023-07-16 09:06:05,544 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,43645,1689498341402 after splitting done 2023-07-16 09:06:05,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7ba9ad3beeb428e3628842526c2eadb2, ASSIGN in 476 msec 2023-07-16 09:06:05,544 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,43645,1689498341402 from processing; numProcessing=0 2023-07-16 09:06:05,546 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,43645,1689498341402, splitWal=true, meta=true in 1.3590 sec 2023-07-16 09:06:05,725 DEBUG [Listener at localhost/37403] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52db3f80 to 127.0.0.1:53764 2023-07-16 09:06:05,725 DEBUG [Listener at localhost/37403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:06:05,725 INFO [Listener at localhost/37403] client.TestScannerTimeout(124): END ************ test2772 2023-07-16 09:06:05,739 INFO [Listener at localhost/37403] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 297), OpenFileDescriptor=577 (was 574) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=250 (was 246) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6133 (was 6261) 2023-07-16 09:06:05,739 INFO [Listener at localhost/37403] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-16 09:06:05,739 INFO [Listener at localhost/37403] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-16 09:06:05,739 DEBUG [Listener at localhost/37403] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61901cae to 127.0.0.1:53764 2023-07-16 09:06:05,739 DEBUG [Listener at localhost/37403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:06:05,740 DEBUG [Listener at localhost/37403] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-16 09:06:05,740 DEBUG [Listener at localhost/37403] util.JVMClusterUtil(257): Found active master hash=1752664830, stopped=false 2023-07-16 09:06:05,740 INFO [Listener at localhost/37403] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:06:05,742 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-16 09:06:05,742 INFO [Listener at localhost/37403] procedure2.ProcedureExecutor(629): Stopping 2023-07-16 09:06:05,742 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-16 09:06:05,742 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:06:05,743 DEBUG [Listener at localhost/37403] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x636dce05 to 127.0.0.1:53764 2023-07-16 09:06:05,743 DEBUG [Listener at localhost/37403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:06:05,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-16 09:06:05,743 INFO [Listener at localhost/37403] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,36585,1689498346638' ***** 2023-07-16 09:06:05,743 INFO [Listener at localhost/37403] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-16 09:06:05,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-16 09:06:05,743 INFO [RS:2;jenkins-hbase4:36585] regionserver.HeapMemoryManager(220): Stopping 2023-07-16 09:06:05,743 INFO [RS:2;jenkins-hbase4:36585] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-16 09:06:05,743 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-16 09:06:05,743 INFO [RS:2;jenkins-hbase4:36585] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-16 09:06:05,744 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(3305): Received CLOSE for f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:05,744 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(3305): Received CLOSE for 7ba9ad3beeb428e3628842526c2eadb2 2023-07-16 09:06:05,744 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:05,744 DEBUG [RS:2;jenkins-hbase4:36585] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c4f7e07 to 127.0.0.1:53764 2023-07-16 09:06:05,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f14b930b48c75cc2e0ecf1557d0ca60d, disabling compactions & flushes 2023-07-16 09:06:05,744 DEBUG [RS:2;jenkins-hbase4:36585] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:06:05,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:05,744 INFO [RS:2;jenkins-hbase4:36585] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-16 09:06:05,745 INFO [RS:2;jenkins-hbase4:36585] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-16 09:06:05,745 INFO [RS:2;jenkins-hbase4:36585] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-16 09:06:05,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:05,745 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-16 09:06:05,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. after waiting 0 ms 2023-07-16 09:06:05,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:05,745 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-16 09:06:05,745 DEBUG [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, f14b930b48c75cc2e0ecf1557d0ca60d=t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d., 7ba9ad3beeb428e3628842526c2eadb2=hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2.} 2023-07-16 09:06:05,745 DEBUG [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1504): Waiting on 1588230740, 7ba9ad3beeb428e3628842526c2eadb2, f14b930b48c75cc2e0ecf1557d0ca60d 2023-07-16 09:06:05,745 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-16 09:06:05,745 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-16 09:06:05,745 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-16 09:06:05,746 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-16 09:06:05,746 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-16 09:06:05,747 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-16 09:06:05,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/default/t/f14b930b48c75cc2e0ecf1557d0ca60d/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-16 09:06:05,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:05,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f14b930b48c75cc2e0ecf1557d0ca60d: 2023-07-16 09:06:05,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689498344091.f14b930b48c75cc2e0ecf1557d0ca60d. 2023-07-16 09:06:05,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7ba9ad3beeb428e3628842526c2eadb2, disabling compactions & flushes 2023-07-16 09:06:05,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:05,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:05,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. after waiting 0 ms 2023-07-16 09:06:05,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:05,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/namespace/7ba9ad3beeb428e3628842526c2eadb2/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-16 09:06:05,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:05,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7ba9ad3beeb428e3628842526c2eadb2: 2023-07-16 09:06:05,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689498343224.7ba9ad3beeb428e3628842526c2eadb2. 2023-07-16 09:06:05,773 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=26 (bloomFilter=false), to=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/.tmp/info/6d8c7a9ad1da4aada1c71bdc34d4156c 2023-07-16 09:06:05,780 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/.tmp/info/6d8c7a9ad1da4aada1c71bdc34d4156c as hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info/6d8c7a9ad1da4aada1c71bdc34d4156c 2023-07-16 09:06:05,786 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/info/6d8c7a9ad1da4aada1c71bdc34d4156c, entries=16, sequenceid=26, filesize=6.4 K 2023-07-16 09:06:05,787 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 41ms, sequenceid=26, compaction requested=true 2023-07-16 09:06:05,794 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/data/hbase/meta/1588230740/recovered.edits/29.seqid, newMaxSeqId=29, maxSeqId=19 2023-07-16 09:06:05,795 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-16 09:06:05,795 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-16 09:06:05,795 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-16 09:06:05,795 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-16 09:06:05,945 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,36585,1689498346638; all regions closed. 2023-07-16 09:06:05,951 DEBUG [RS:2;jenkins-hbase4:36585] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs 2023-07-16 09:06:05,951 INFO [RS:2;jenkins-hbase4:36585] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36585%2C1689498346638.meta:.meta(num 1689498364738) 2023-07-16 09:06:05,957 DEBUG [RS:2;jenkins-hbase4:36585] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/oldWALs 2023-07-16 09:06:05,957 INFO [RS:2;jenkins-hbase4:36585] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36585%2C1689498346638:(num 1689498346826) 2023-07-16 09:06:05,957 DEBUG [RS:2;jenkins-hbase4:36585] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:06:05,957 INFO [RS:2;jenkins-hbase4:36585] regionserver.LeaseManager(133): Closed leases 2023-07-16 09:06:05,957 INFO [RS:2;jenkins-hbase4:36585] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-16 09:06:05,957 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-16 09:06:05,958 INFO [RS:2;jenkins-hbase4:36585] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36585 2023-07-16 09:06:05,960 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:06:05,960 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36585,1689498346638 2023-07-16 09:06:05,960 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-16 09:06:05,961 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36585,1689498346638] 2023-07-16 09:06:05,961 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36585,1689498346638; numProcessing=1 2023-07-16 09:06:05,965 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36585,1689498346638 already deleted, retry=false 2023-07-16 09:06:05,965 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36585,1689498346638 expired; onlineServers=0 2023-07-16 09:06:05,965 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,46859,1689498340206' ***** 2023-07-16 09:06:05,965 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-16 09:06:05,965 DEBUG [M:0;jenkins-hbase4:46859] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@294a80c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-16 09:06:05,966 INFO [M:0;jenkins-hbase4:46859] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,46859,1689498340206 2023-07-16 09:06:05,966 INFO [M:0;jenkins-hbase4:46859] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,46859,1689498340206; all regions closed. 2023-07-16 09:06:05,966 DEBUG [M:0;jenkins-hbase4:46859] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-16 09:06:05,966 DEBUG [M:0;jenkins-hbase4:46859] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-16 09:06:05,966 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-16 09:06:05,966 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689498342424] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689498342424,5,FailOnTimeoutGroup] 2023-07-16 09:06:05,966 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689498342424] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689498342424,5,FailOnTimeoutGroup] 2023-07-16 09:06:05,966 DEBUG [M:0;jenkins-hbase4:46859] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-16 09:06:05,967 INFO [M:0;jenkins-hbase4:46859] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-16 09:06:05,967 INFO [M:0;jenkins-hbase4:46859] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-16 09:06:05,967 INFO [M:0;jenkins-hbase4:46859] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-16 09:06:05,967 DEBUG [M:0;jenkins-hbase4:46859] master.HMaster(1512): Stopping service threads 2023-07-16 09:06:05,967 INFO [M:0;jenkins-hbase4:46859] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-16 09:06:05,967 ERROR [M:0;jenkins-hbase4:46859] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1320148163) connection to localhost/127.0.0.1:36827 from jenkins,5,PEWorkerGroup] 2023-07-16 09:06:05,968 INFO [M:0;jenkins-hbase4:46859] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-16 09:06:05,968 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-16 09:06:05,968 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-16 09:06:05,968 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-16 09:06:05,968 DEBUG [M:0;jenkins-hbase4:46859] zookeeper.ZKUtil(398): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-16 09:06:05,968 WARN [M:0;jenkins-hbase4:46859] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-16 09:06:05,968 INFO [M:0;jenkins-hbase4:46859] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-16 09:06:05,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-16 09:06:05,969 INFO [M:0;jenkins-hbase4:46859] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-16 09:06:05,969 DEBUG [M:0;jenkins-hbase4:46859] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-16 09:06:05,969 INFO [M:0;jenkins-hbase4:46859] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-16 09:06:05,969 DEBUG [M:0;jenkins-hbase4:46859] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-16 09:06:05,969 DEBUG [M:0;jenkins-hbase4:46859] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-16 09:06:05,969 DEBUG [M:0;jenkins-hbase4:46859] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-16 09:06:05,969 INFO [M:0;jenkins-hbase4:46859] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=111.31 KB heapSize=134.27 KB 2023-07-16 09:06:05,985 INFO [M:0;jenkins-hbase4:46859] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=111.31 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/504d781a69b44de1840713ace48a7c52 2023-07-16 09:06:05,992 DEBUG [M:0;jenkins-hbase4:46859] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/504d781a69b44de1840713ace48a7c52 as hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/504d781a69b44de1840713ace48a7c52 2023-07-16 09:06:06,003 INFO [M:0;jenkins-hbase4:46859] regionserver.HStore(1080): Added hdfs://localhost:36827/user/jenkins/test-data/797f7ab7-c450-c2c4-0c29-4fcd6c9a2e76/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/504d781a69b44de1840713ace48a7c52, entries=33, sequenceid=257, filesize=8.5 K 2023-07-16 09:06:06,004 INFO [M:0;jenkins-hbase4:46859] regionserver.HRegion(2948): Finished flush of dataSize ~111.31 KB/113983, heapSize ~134.26 KB/137480, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 35ms, sequenceid=257, compaction requested=false 2023-07-16 09:06:06,007 INFO [M:0;jenkins-hbase4:46859] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-16 09:06:06,007 DEBUG [M:0;jenkins-hbase4:46859] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-16 09:06:06,013 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-16 09:06:06,013 INFO [M:0;jenkins-hbase4:46859] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-16 09:06:06,013 INFO [M:0;jenkins-hbase4:46859] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46859 2023-07-16 09:06:06,015 DEBUG [M:0;jenkins-hbase4:46859] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46859,1689498340206 already deleted, retry=false 2023-07-16 09:06:06,230 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-16 09:06:06,230 INFO [M:0;jenkins-hbase4:46859] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,46859,1689498340206; zookeeper connection closed. 2023-07-16 09:06:06,230 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): master:46859-0x1016d614b290000, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-16 09:06:06,330 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-16 09:06:06,330 INFO [RS:2;jenkins-hbase4:36585] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,36585,1689498346638; zookeeper connection closed. 2023-07-16 09:06:06,330 DEBUG [Listener at localhost/37403-EventThread] zookeeper.ZKWatcher(600): regionserver:36585-0x1016d614b290008, quorum=127.0.0.1:53764, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-16 09:06:06,331 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4592f6c5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4592f6c5 2023-07-16 09:06:06,331 INFO [Listener at localhost/37403] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-16 09:06:06,331 WARN [Listener at localhost/37403] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-16 09:06:06,334 INFO [Listener at localhost/37403] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-16 09:06:06,440 WARN [BP-887411987-172.31.14.131-1689498337264 heartbeating to localhost/127.0.0.1:36827] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-16 09:06:06,440 WARN [BP-887411987-172.31.14.131-1689498337264 heartbeating to localhost/127.0.0.1:36827] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-887411987-172.31.14.131-1689498337264 (Datanode Uuid 64a861ed-56f4-41e6-abbb-eb71c4f0710a) service to localhost/127.0.0.1:36827 2023-07-16 09:06:06,441 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/dfs/data/data3/current/BP-887411987-172.31.14.131-1689498337264] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-16 09:06:06,442 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/dfs/data/data4/current/BP-887411987-172.31.14.131-1689498337264] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-16 09:06:06,443 WARN [Listener at localhost/37403] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-16 09:06:06,447 INFO [Listener at localhost/37403] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-16 09:06:06,552 WARN [BP-887411987-172.31.14.131-1689498337264 heartbeating to localhost/127.0.0.1:36827] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-16 09:06:06,552 WARN [BP-887411987-172.31.14.131-1689498337264 heartbeating to localhost/127.0.0.1:36827] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-887411987-172.31.14.131-1689498337264 (Datanode Uuid aa48e06f-2444-4a8e-b26c-1906e3c7be6b) service to localhost/127.0.0.1:36827 2023-07-16 09:06:06,552 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/dfs/data/data1/current/BP-887411987-172.31.14.131-1689498337264] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-16 09:06:06,553 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/05cdf329-27a5-d97d-8c47-21d69cc12f50/cluster_76d6b534-c4be-c62a-713f-11cc3fe3519f/dfs/data/data2/current/BP-887411987-172.31.14.131-1689498337264] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-16 09:06:06,599 INFO [Listener at localhost/37403] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-16 09:06:06,697 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-16 09:06:06,711 INFO [Listener at localhost/37403] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-16 09:06:06,745 INFO [Listener at localhost/37403] hbase.HBaseTestingUtility(1293): Minicluster is down