2023-07-19 13:05:23,240 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-19 13:05:23,486 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e 2023-07-19 13:05:23,499 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-19 13:05:23,499 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1, deleteOnExit=true 2023-07-19 13:05:23,499 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-19 13:05:23,500 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/test.cache.data in system properties and HBase conf 2023-07-19 13:05:23,500 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/hadoop.tmp.dir in system properties and HBase conf 2023-07-19 13:05:23,501 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/hadoop.log.dir in system properties and HBase conf 2023-07-19 13:05:23,501 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-19 13:05:23,502 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-19 13:05:23,502 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-19 13:05:23,618 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-19 13:05:24,049 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-19 13:05:24,054 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-19 13:05:24,055 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-19 13:05:24,055 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-19 13:05:24,056 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-19 13:05:24,056 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-19 13:05:24,057 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-19 13:05:24,057 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-19 13:05:24,058 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-19 13:05:24,058 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-19 13:05:24,059 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/nfs.dump.dir in system properties and HBase conf 2023-07-19 13:05:24,059 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/java.io.tmpdir in system properties and HBase conf 2023-07-19 13:05:24,059 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-19 13:05:24,060 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-19 13:05:24,060 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-19 13:05:24,626 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-19 13:05:24,630 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-19 13:05:24,910 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-19 13:05:25,085 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-19 13:05:25,101 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 13:05:25,139 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 13:05:25,176 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/java.io.tmpdir/Jetty_localhost_38717_hdfs____148fr6/webapp 2023-07-19 13:05:25,358 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38717 2023-07-19 13:05:25,368 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-19 13:05:25,368 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-19 13:05:25,768 WARN [Listener at localhost/33275] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 13:05:25,869 WARN [Listener at localhost/33275] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-19 13:05:25,886 WARN [Listener at localhost/33275] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 13:05:25,892 INFO [Listener at localhost/33275] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 13:05:25,896 INFO [Listener at localhost/33275] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/java.io.tmpdir/Jetty_localhost_33301_datanode____zdetx7/webapp 2023-07-19 13:05:25,991 INFO [Listener at localhost/33275] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33301 2023-07-19 13:05:26,294 WARN [Listener at localhost/39227] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 13:05:26,315 WARN [Listener at localhost/39227] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-19 13:05:26,319 WARN [Listener at localhost/39227] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-19 13:05:26,321 INFO [Listener at localhost/39227] log.Slf4jLog(67): jetty-6.1.26 2023-07-19 13:05:26,329 INFO [Listener at localhost/39227] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/java.io.tmpdir/Jetty_localhost_37599_datanode____8fcwig/webapp 2023-07-19 13:05:26,429 INFO [Listener at localhost/39227] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37599 2023-07-19 13:05:26,438 WARN [Listener at localhost/39077] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-19 13:05:26,789 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x13198e1b462430b2: Processing first storage report for DS-d790c55a-9d91-427b-ad7b-0a1912fa7255 from datanode 50b8eed1-fa26-49b1-8770-360fc1e33917 2023-07-19 13:05:26,791 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x13198e1b462430b2: from storage DS-d790c55a-9d91-427b-ad7b-0a1912fa7255 node DatanodeRegistration(127.0.0.1:33723, datanodeUuid=50b8eed1-fa26-49b1-8770-360fc1e33917, infoPort=43267, infoSecurePort=0, ipcPort=39227, storageInfo=lv=-57;cid=testClusterID;nsid=750551982;c=1689771924702), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-19 13:05:26,791 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb56211c19d000890: Processing first storage report for DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007 from datanode 6fb50309-88be-4da3-adde-59fbc3578eed 2023-07-19 13:05:26,791 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb56211c19d000890: from storage DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007 node DatanodeRegistration(127.0.0.1:35447, datanodeUuid=6fb50309-88be-4da3-adde-59fbc3578eed, infoPort=41613, infoSecurePort=0, ipcPort=39077, storageInfo=lv=-57;cid=testClusterID;nsid=750551982;c=1689771924702), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 13:05:26,792 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x13198e1b462430b2: Processing first storage report for DS-7532a2e7-fc02-4503-ae88-9a3ed215d979 from datanode 50b8eed1-fa26-49b1-8770-360fc1e33917 2023-07-19 13:05:26,792 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x13198e1b462430b2: from storage DS-7532a2e7-fc02-4503-ae88-9a3ed215d979 node DatanodeRegistration(127.0.0.1:33723, datanodeUuid=50b8eed1-fa26-49b1-8770-360fc1e33917, infoPort=43267, infoSecurePort=0, ipcPort=39227, storageInfo=lv=-57;cid=testClusterID;nsid=750551982;c=1689771924702), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 13:05:26,792 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb56211c19d000890: Processing first storage report for DS-d2269151-9097-437a-98c1-42cbe6c1bc79 from datanode 6fb50309-88be-4da3-adde-59fbc3578eed 2023-07-19 13:05:26,792 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb56211c19d000890: from storage DS-d2269151-9097-437a-98c1-42cbe6c1bc79 node DatanodeRegistration(127.0.0.1:35447, datanodeUuid=6fb50309-88be-4da3-adde-59fbc3578eed, infoPort=41613, infoSecurePort=0, ipcPort=39077, storageInfo=lv=-57;cid=testClusterID;nsid=750551982;c=1689771924702), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-19 13:05:26,896 DEBUG [Listener at localhost/39077] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e 2023-07-19 13:05:26,965 INFO [Listener at localhost/39077] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1/zookeeper_0, clientPort=58251, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-19 13:05:26,979 INFO [Listener at localhost/39077] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58251 2023-07-19 13:05:26,987 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:26,989 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:27,645 INFO [Listener at localhost/39077] util.FSUtils(471): Created version file at hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5 with version=8 2023-07-19 13:05:27,645 INFO [Listener at localhost/39077] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/hbase-staging 2023-07-19 13:05:28,007 INFO [Listener at localhost/39077] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-19 13:05:28,583 INFO [Listener at localhost/39077] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 13:05:28,620 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:28,621 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:28,621 INFO [Listener at localhost/39077] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 13:05:28,621 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:28,621 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 13:05:28,777 INFO [Listener at localhost/39077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 13:05:28,846 DEBUG [Listener at localhost/39077] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-19 13:05:28,939 INFO [Listener at localhost/39077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40839 2023-07-19 13:05:28,949 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:28,951 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:28,971 INFO [Listener at localhost/39077] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40839 connecting to ZooKeeper ensemble=127.0.0.1:58251 2023-07-19 13:05:29,013 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:408390x0, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 13:05:29,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:40839-0x1017dafe8aa0000 connected 2023-07-19 13:05:29,039 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 13:05:29,040 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 13:05:29,044 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 13:05:29,052 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40839 2023-07-19 13:05:29,052 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40839 2023-07-19 13:05:29,052 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40839 2023-07-19 13:05:29,053 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40839 2023-07-19 13:05:29,053 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40839 2023-07-19 13:05:29,059 INFO [Listener at localhost/39077] master.HMaster(444): hbase.rootdir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5, hbase.cluster.distributed=false 2023-07-19 13:05:29,132 INFO [Listener at localhost/39077] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 13:05:29,133 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:29,133 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:29,133 INFO [Listener at localhost/39077] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 13:05:29,133 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:29,133 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 13:05:29,138 INFO [Listener at localhost/39077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 13:05:29,142 INFO [Listener at localhost/39077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39017 2023-07-19 13:05:29,144 INFO [Listener at localhost/39077] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 13:05:29,150 DEBUG [Listener at localhost/39077] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 13:05:29,152 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:29,154 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:29,156 INFO [Listener at localhost/39077] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39017 connecting to ZooKeeper ensemble=127.0.0.1:58251 2023-07-19 13:05:29,160 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:390170x0, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 13:05:29,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39017-0x1017dafe8aa0001 connected 2023-07-19 13:05:29,161 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 13:05:29,162 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 13:05:29,163 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 13:05:29,164 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39017 2023-07-19 13:05:29,164 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39017 2023-07-19 13:05:29,164 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39017 2023-07-19 13:05:29,165 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39017 2023-07-19 13:05:29,165 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39017 2023-07-19 13:05:29,178 INFO [Listener at localhost/39077] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 13:05:29,179 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:29,179 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:29,179 INFO [Listener at localhost/39077] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 13:05:29,179 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:29,180 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 13:05:29,180 INFO [Listener at localhost/39077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 13:05:29,181 INFO [Listener at localhost/39077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36157 2023-07-19 13:05:29,181 INFO [Listener at localhost/39077] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 13:05:29,183 DEBUG [Listener at localhost/39077] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 13:05:29,183 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:29,185 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:29,187 INFO [Listener at localhost/39077] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36157 connecting to ZooKeeper ensemble=127.0.0.1:58251 2023-07-19 13:05:29,190 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:361570x0, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 13:05:29,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36157-0x1017dafe8aa0002 connected 2023-07-19 13:05:29,191 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 13:05:29,192 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 13:05:29,193 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 13:05:29,194 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36157 2023-07-19 13:05:29,194 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36157 2023-07-19 13:05:29,194 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36157 2023-07-19 13:05:29,195 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36157 2023-07-19 13:05:29,195 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36157 2023-07-19 13:05:29,196 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:29,205 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-19 13:05:29,206 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:29,224 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 13:05:29,224 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 13:05:29,224 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-19 13:05:29,225 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:29,226 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 13:05:29,227 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,40839,1689771927809 from backup master directory 2023-07-19 13:05:29,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 13:05:29,231 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:29,231 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-19 13:05:29,232 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 13:05:29,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:29,235 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-19 13:05:29,236 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-19 13:05:29,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/hbase.id with ID: 528e9326-871c-4f3c-b648-fadc18694914 2023-07-19 13:05:29,359 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:29,373 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:29,417 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x788474c0 to 127.0.0.1:58251 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 13:05:29,453 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66a5b845, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 13:05:29,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-19 13:05:29,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-19 13:05:29,498 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-19 13:05:29,499 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-19 13:05:29,501 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-19 13:05:29,506 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-19 13:05:29,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 13:05:29,544 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/data/master/store-tmp 2023-07-19 13:05:29,575 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:29,575 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-19 13:05:29,576 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 13:05:29,576 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 13:05:29,576 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-19 13:05:29,576 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 13:05:29,576 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 13:05:29,576 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 13:05:29,577 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/WALs/jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:29,597 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40839%2C1689771927809, suffix=, logDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/WALs/jenkins-hbase4.apache.org,40839,1689771927809, archiveDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/oldWALs, maxLogs=10 2023-07-19 13:05:29,650 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK] 2023-07-19 13:05:29,650 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK] 2023-07-19 13:05:29,657 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 13:05:29,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/WALs/jenkins-hbase4.apache.org,40839,1689771927809/jenkins-hbase4.apache.org%2C40839%2C1689771927809.1689771929607 2023-07-19 13:05:29,721 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK], DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK]] 2023-07-19 13:05:29,722 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:29,722 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:29,726 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 13:05:29,727 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 13:05:29,788 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-19 13:05:29,796 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-19 13:05:29,821 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-19 13:05:29,832 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:29,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-19 13:05:29,840 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-19 13:05:29,854 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-19 13:05:29,858 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 13:05:29,859 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64800890, jitterRate=-0.03439149260520935}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 13:05:29,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 13:05:29,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-19 13:05:29,879 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-19 13:05:29,879 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-19 13:05:29,881 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-19 13:05:29,883 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-19 13:05:29,915 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-19 13:05:29,915 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-19 13:05:29,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-19 13:05:29,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-19 13:05:29,968 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-19 13:05:29,971 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-19 13:05:29,972 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-19 13:05:29,977 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-19 13:05:29,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-19 13:05:29,984 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:29,985 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-19 13:05:29,985 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-19 13:05:29,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-19 13:05:30,002 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 13:05:30,002 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 13:05:30,002 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-19 13:05:30,002 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:30,003 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,40839,1689771927809, sessionid=0x1017dafe8aa0000, setting cluster-up flag (Was=false) 2023-07-19 13:05:30,017 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:30,022 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-19 13:05:30,024 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:30,028 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:30,035 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-19 13:05:30,036 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:30,038 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.hbase-snapshot/.tmp 2023-07-19 13:05:30,098 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(951): ClusterId : 528e9326-871c-4f3c-b648-fadc18694914 2023-07-19 13:05:30,098 INFO [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(951): ClusterId : 528e9326-871c-4f3c-b648-fadc18694914 2023-07-19 13:05:30,104 DEBUG [RS:1;jenkins-hbase4:36157] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 13:05:30,104 DEBUG [RS:0;jenkins-hbase4:39017] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 13:05:30,110 DEBUG [RS:0;jenkins-hbase4:39017] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 13:05:30,110 DEBUG [RS:1;jenkins-hbase4:36157] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 13:05:30,110 DEBUG [RS:0;jenkins-hbase4:39017] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 13:05:30,111 DEBUG [RS:1;jenkins-hbase4:36157] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 13:05:30,115 DEBUG [RS:0;jenkins-hbase4:39017] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 13:05:30,115 DEBUG [RS:1;jenkins-hbase4:36157] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 13:05:30,116 DEBUG [RS:0;jenkins-hbase4:39017] zookeeper.ReadOnlyZKClient(139): Connect 0x0725691c to 127.0.0.1:58251 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 13:05:30,117 DEBUG [RS:1;jenkins-hbase4:36157] zookeeper.ReadOnlyZKClient(139): Connect 0x5b40a713 to 127.0.0.1:58251 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 13:05:30,122 DEBUG [RS:0;jenkins-hbase4:39017] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9e29788, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 13:05:30,127 DEBUG [RS:1;jenkins-hbase4:36157] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5587520b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 13:05:30,127 DEBUG [RS:0;jenkins-hbase4:39017] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ddd026c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 13:05:30,127 DEBUG [RS:1;jenkins-hbase4:36157] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8cc63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 13:05:30,154 DEBUG [RS:0;jenkins-hbase4:39017] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:39017 2023-07-19 13:05:30,154 DEBUG [RS:1;jenkins-hbase4:36157] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:36157 2023-07-19 13:05:30,158 INFO [RS:0;jenkins-hbase4:39017] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 13:05:30,158 INFO [RS:1;jenkins-hbase4:36157] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 13:05:30,159 INFO [RS:1;jenkins-hbase4:36157] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 13:05:30,159 INFO [RS:0;jenkins-hbase4:39017] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 13:05:30,159 DEBUG [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 13:05:30,159 DEBUG [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 13:05:30,162 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,40839,1689771927809 with isa=jenkins-hbase4.apache.org/172.31.14.131:36157, startcode=1689771929178 2023-07-19 13:05:30,162 INFO [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,40839,1689771927809 with isa=jenkins-hbase4.apache.org/172.31.14.131:39017, startcode=1689771929131 2023-07-19 13:05:30,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-19 13:05:30,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 13:05:30,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 13:05:30,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 13:05:30,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-19 13:05:30,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-19 13:05:30,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 13:05:30,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,179 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689771960179 2023-07-19 13:05:30,180 DEBUG [RS:1;jenkins-hbase4:36157] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 13:05:30,180 DEBUG [RS:0;jenkins-hbase4:39017] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 13:05:30,181 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-19 13:05:30,185 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-19 13:05:30,186 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-19 13:05:30,192 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-19 13:05:30,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-19 13:05:30,202 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-19 13:05:30,203 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-19 13:05:30,203 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-19 13:05:30,203 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-19 13:05:30,204 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-19 13:05:30,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-19 13:05:30,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-19 13:05:30,214 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-19 13:05:30,214 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-19 13:05:30,217 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689771930217,5,FailOnTimeoutGroup] 2023-07-19 13:05:30,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689771930218,5,FailOnTimeoutGroup] 2023-07-19 13:05:30,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,219 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-19 13:05:30,220 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,220 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,236 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-19 13:05:30,237 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-19 13:05:30,238 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5 2023-07-19 13:05:30,249 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60809, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 13:05:30,249 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38803, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 13:05:30,260 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:30,263 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 13:05:30,265 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40839] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:30,266 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/info 2023-07-19 13:05:30,267 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 13:05:30,268 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:30,268 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 13:05:30,271 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/rep_barrier 2023-07-19 13:05:30,271 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 13:05:30,272 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:30,273 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 13:05:30,275 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/table 2023-07-19 13:05:30,276 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40839] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:30,277 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 13:05:30,278 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:30,279 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740 2023-07-19 13:05:30,280 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740 2023-07-19 13:05:30,283 DEBUG [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5 2023-07-19 13:05:30,283 DEBUG [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5 2023-07-19 13:05:30,283 DEBUG [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:33275 2023-07-19 13:05:30,283 DEBUG [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:33275 2023-07-19 13:05:30,283 DEBUG [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 13:05:30,283 DEBUG [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 13:05:30,285 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 13:05:30,287 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 13:05:30,288 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:30,292 DEBUG [RS:0;jenkins-hbase4:39017] zookeeper.ZKUtil(162): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:30,292 DEBUG [RS:1;jenkins-hbase4:36157] zookeeper.ZKUtil(162): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:30,292 WARN [RS:0;jenkins-hbase4:39017] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 13:05:30,292 WARN [RS:1;jenkins-hbase4:36157] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 13:05:30,295 INFO [RS:0;jenkins-hbase4:39017] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 13:05:30,296 INFO [RS:1;jenkins-hbase4:36157] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 13:05:30,296 DEBUG [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1948): logDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:30,296 DEBUG [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1948): logDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:30,297 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 13:05:30,297 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36157,1689771929178] 2023-07-19 13:05:30,297 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39017,1689771929131] 2023-07-19 13:05:30,299 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60672178, jitterRate=-0.0959140956401825}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 13:05:30,299 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 13:05:30,300 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 13:05:30,300 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 13:05:30,300 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 13:05:30,300 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 13:05:30,300 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 13:05:30,302 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 13:05:30,302 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 13:05:30,307 DEBUG [RS:1;jenkins-hbase4:36157] zookeeper.ZKUtil(162): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:30,307 DEBUG [RS:0;jenkins-hbase4:39017] zookeeper.ZKUtil(162): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:30,308 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-19 13:05:30,308 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-19 13:05:30,308 DEBUG [RS:1;jenkins-hbase4:36157] zookeeper.ZKUtil(162): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:30,308 DEBUG [RS:0;jenkins-hbase4:39017] zookeeper.ZKUtil(162): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:30,317 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-19 13:05:30,323 DEBUG [RS:0;jenkins-hbase4:39017] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 13:05:30,323 DEBUG [RS:1;jenkins-hbase4:36157] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 13:05:30,332 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-19 13:05:30,333 INFO [RS:1;jenkins-hbase4:36157] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 13:05:30,333 INFO [RS:0;jenkins-hbase4:39017] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 13:05:30,335 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-19 13:05:30,353 INFO [RS:1;jenkins-hbase4:36157] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 13:05:30,353 INFO [RS:0;jenkins-hbase4:39017] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 13:05:30,357 INFO [RS:1;jenkins-hbase4:36157] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 13:05:30,357 INFO [RS:0;jenkins-hbase4:39017] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 13:05:30,357 INFO [RS:1;jenkins-hbase4:36157] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,357 INFO [RS:0;jenkins-hbase4:39017] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,358 INFO [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 13:05:30,358 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 13:05:30,366 INFO [RS:0;jenkins-hbase4:39017] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,366 INFO [RS:1;jenkins-hbase4:36157] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,366 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,366 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 13:05:30,367 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 13:05:30,367 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,367 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,368 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,368 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,368 DEBUG [RS:0;jenkins-hbase4:39017] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,368 DEBUG [RS:1;jenkins-hbase4:36157] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:30,369 INFO [RS:0;jenkins-hbase4:39017] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,369 INFO [RS:1;jenkins-hbase4:36157] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,369 INFO [RS:0;jenkins-hbase4:39017] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,369 INFO [RS:1;jenkins-hbase4:36157] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,369 INFO [RS:0;jenkins-hbase4:39017] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,369 INFO [RS:1;jenkins-hbase4:36157] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,386 INFO [RS:1;jenkins-hbase4:36157] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 13:05:30,386 INFO [RS:0;jenkins-hbase4:39017] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 13:05:30,389 INFO [RS:0;jenkins-hbase4:39017] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39017,1689771929131-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,389 INFO [RS:1;jenkins-hbase4:36157] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36157,1689771929178-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:30,409 INFO [RS:1;jenkins-hbase4:36157] regionserver.Replication(203): jenkins-hbase4.apache.org,36157,1689771929178 started 2023-07-19 13:05:30,409 INFO [RS:0;jenkins-hbase4:39017] regionserver.Replication(203): jenkins-hbase4.apache.org,39017,1689771929131 started 2023-07-19 13:05:30,409 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,36157,1689771929178, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36157, sessionid=0x1017dafe8aa0002 2023-07-19 13:05:30,409 INFO [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,39017,1689771929131, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39017, sessionid=0x1017dafe8aa0001 2023-07-19 13:05:30,409 DEBUG [RS:0;jenkins-hbase4:39017] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 13:05:30,409 DEBUG [RS:1;jenkins-hbase4:36157] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 13:05:30,409 DEBUG [RS:0;jenkins-hbase4:39017] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:30,409 DEBUG [RS:1;jenkins-hbase4:36157] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:30,410 DEBUG [RS:0;jenkins-hbase4:39017] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39017,1689771929131' 2023-07-19 13:05:30,410 DEBUG [RS:1;jenkins-hbase4:36157] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36157,1689771929178' 2023-07-19 13:05:30,410 DEBUG [RS:1;jenkins-hbase4:36157] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 13:05:30,410 DEBUG [RS:0;jenkins-hbase4:39017] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 13:05:30,411 DEBUG [RS:0;jenkins-hbase4:39017] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 13:05:30,411 DEBUG [RS:1;jenkins-hbase4:36157] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 13:05:30,412 DEBUG [RS:0;jenkins-hbase4:39017] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 13:05:30,412 DEBUG [RS:0;jenkins-hbase4:39017] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 13:05:30,412 DEBUG [RS:1;jenkins-hbase4:36157] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 13:05:30,412 DEBUG [RS:0;jenkins-hbase4:39017] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:30,412 DEBUG [RS:1;jenkins-hbase4:36157] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 13:05:30,412 DEBUG [RS:0;jenkins-hbase4:39017] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39017,1689771929131' 2023-07-19 13:05:30,413 DEBUG [RS:0;jenkins-hbase4:39017] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 13:05:30,412 DEBUG [RS:1;jenkins-hbase4:36157] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:30,413 DEBUG [RS:1;jenkins-hbase4:36157] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36157,1689771929178' 2023-07-19 13:05:30,413 DEBUG [RS:1;jenkins-hbase4:36157] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 13:05:30,413 DEBUG [RS:0;jenkins-hbase4:39017] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 13:05:30,413 DEBUG [RS:1;jenkins-hbase4:36157] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 13:05:30,414 DEBUG [RS:1;jenkins-hbase4:36157] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 13:05:30,414 DEBUG [RS:0;jenkins-hbase4:39017] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 13:05:30,414 INFO [RS:1;jenkins-hbase4:36157] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 13:05:30,414 INFO [RS:0;jenkins-hbase4:39017] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 13:05:30,414 INFO [RS:1;jenkins-hbase4:36157] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 13:05:30,414 INFO [RS:0;jenkins-hbase4:39017] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 13:05:30,487 DEBUG [jenkins-hbase4:40839] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-19 13:05:30,490 DEBUG [jenkins-hbase4:40839] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 13:05:30,497 DEBUG [jenkins-hbase4:40839] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 13:05:30,497 DEBUG [jenkins-hbase4:40839] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 13:05:30,497 DEBUG [jenkins-hbase4:40839] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 13:05:30,499 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36157,1689771929178, state=OPENING 2023-07-19 13:05:30,506 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-19 13:05:30,508 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:30,508 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 13:05:30,512 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36157,1689771929178}] 2023-07-19 13:05:30,526 INFO [RS:0;jenkins-hbase4:39017] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39017%2C1689771929131, suffix=, logDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131, archiveDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs, maxLogs=32 2023-07-19 13:05:30,526 INFO [RS:1;jenkins-hbase4:36157] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36157%2C1689771929178, suffix=, logDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178, archiveDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs, maxLogs=32 2023-07-19 13:05:30,549 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK] 2023-07-19 13:05:30,549 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK] 2023-07-19 13:05:30,549 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK] 2023-07-19 13:05:30,550 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK] 2023-07-19 13:05:30,560 INFO [RS:0;jenkins-hbase4:39017] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530 2023-07-19 13:05:30,560 INFO [RS:1;jenkins-hbase4:36157] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530 2023-07-19 13:05:30,560 DEBUG [RS:0;jenkins-hbase4:39017] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK], DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK]] 2023-07-19 13:05:30,560 DEBUG [RS:1;jenkins-hbase4:36157] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK], DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK]] 2023-07-19 13:05:30,698 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:30,700 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 13:05:30,703 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47948, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 13:05:30,714 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-19 13:05:30,714 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 13:05:30,717 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta, suffix=.meta, logDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178, archiveDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs, maxLogs=32 2023-07-19 13:05:30,734 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK] 2023-07-19 13:05:30,734 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK] 2023-07-19 13:05:30,741 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta 2023-07-19 13:05:30,742 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK], DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK]] 2023-07-19 13:05:30,742 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:30,744 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-19 13:05:30,768 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-19 13:05:30,773 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-19 13:05:30,779 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-19 13:05:30,779 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:30,779 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-19 13:05:30,779 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-19 13:05:30,782 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 13:05:30,784 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/info 2023-07-19 13:05:30,785 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/info 2023-07-19 13:05:30,785 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 13:05:30,787 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:30,787 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 13:05:30,790 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/rep_barrier 2023-07-19 13:05:30,790 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/rep_barrier 2023-07-19 13:05:30,791 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 13:05:30,792 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:30,792 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 13:05:30,794 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/table 2023-07-19 13:05:30,794 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/table 2023-07-19 13:05:30,794 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 13:05:30,795 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:30,797 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740 2023-07-19 13:05:30,800 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740 2023-07-19 13:05:30,803 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 13:05:30,805 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 13:05:30,807 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68754517, jitterRate=0.024522140622138977}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 13:05:30,807 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 13:05:30,816 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689771930690 2023-07-19 13:05:30,834 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-19 13:05:30,834 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-19 13:05:30,835 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36157,1689771929178, state=OPEN 2023-07-19 13:05:30,838 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 13:05:30,838 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 13:05:30,842 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-19 13:05:30,842 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36157,1689771929178 in 326 msec 2023-07-19 13:05:30,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-19 13:05:30,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 527 msec 2023-07-19 13:05:30,852 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 770 msec 2023-07-19 13:05:30,852 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689771930852, completionTime=-1 2023-07-19 13:05:30,852 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-19 13:05:30,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-19 13:05:30,920 DEBUG [hconnection-0x243edf0a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 13:05:30,923 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47962, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 13:05:30,984 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-19 13:05:30,984 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689771990984 2023-07-19 13:05:30,984 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689772050984 2023-07-19 13:05:30,984 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 131 msec 2023-07-19 13:05:31,009 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40839,1689771927809-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:31,009 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40839,1689771927809-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:31,010 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40839,1689771927809-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:31,012 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:40839, period=300000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:31,012 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:31,018 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-19 13:05:31,027 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-19 13:05:31,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-19 13:05:31,040 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-19 13:05:31,042 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-19 13:05:31,046 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-19 13:05:31,072 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:31,075 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38 empty. 2023-07-19 13:05:31,076 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:31,076 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-19 13:05:31,107 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-19 13:05:31,110 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1376f420d426ab5ccf6d1849d6a24f38, NAME => 'hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp 2023-07-19 13:05:31,130 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:31,130 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 1376f420d426ab5ccf6d1849d6a24f38, disabling compactions & flushes 2023-07-19 13:05:31,130 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:31,131 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:31,131 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. after waiting 0 ms 2023-07-19 13:05:31,131 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:31,131 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:31,131 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 1376f420d426ab5ccf6d1849d6a24f38: 2023-07-19 13:05:31,136 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-19 13:05:31,152 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689771931138"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689771931138"}]},"ts":"1689771931138"} 2023-07-19 13:05:31,179 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-19 13:05:31,181 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-19 13:05:31,186 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689771931181"}]},"ts":"1689771931181"} 2023-07-19 13:05:31,192 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-19 13:05:31,197 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 13:05:31,198 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 13:05:31,198 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 13:05:31,198 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 13:05:31,201 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1376f420d426ab5ccf6d1849d6a24f38, ASSIGN}] 2023-07-19 13:05:31,204 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1376f420d426ab5ccf6d1849d6a24f38, ASSIGN 2023-07-19 13:05:31,207 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=1376f420d426ab5ccf6d1849d6a24f38, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36157,1689771929178; forceNewPlan=false, retain=false 2023-07-19 13:05:31,360 INFO [jenkins-hbase4:40839] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-19 13:05:31,361 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1376f420d426ab5ccf6d1849d6a24f38, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:31,361 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689771931361"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689771931361"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689771931361"}]},"ts":"1689771931361"} 2023-07-19 13:05:31,367 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 1376f420d426ab5ccf6d1849d6a24f38, server=jenkins-hbase4.apache.org,36157,1689771929178}] 2023-07-19 13:05:31,529 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:31,529 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1376f420d426ab5ccf6d1849d6a24f38, NAME => 'hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:31,530 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:31,530 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:31,530 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:31,530 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:31,533 INFO [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:31,536 DEBUG [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/info 2023-07-19 13:05:31,536 DEBUG [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/info 2023-07-19 13:05:31,536 INFO [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1376f420d426ab5ccf6d1849d6a24f38 columnFamilyName info 2023-07-19 13:05:31,537 INFO [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] regionserver.HStore(310): Store=1376f420d426ab5ccf6d1849d6a24f38/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:31,539 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:31,540 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:31,545 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:31,551 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 13:05:31,552 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1376f420d426ab5ccf6d1849d6a24f38; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73325656, jitterRate=0.09263741970062256}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 13:05:31,552 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1376f420d426ab5ccf6d1849d6a24f38: 2023-07-19 13:05:31,555 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38., pid=6, masterSystemTime=1689771931521 2023-07-19 13:05:31,560 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:31,560 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:31,562 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1376f420d426ab5ccf6d1849d6a24f38, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:31,562 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689771931561"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689771931561"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689771931561"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689771931561"}]},"ts":"1689771931561"} 2023-07-19 13:05:31,572 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-19 13:05:31,572 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 1376f420d426ab5ccf6d1849d6a24f38, server=jenkins-hbase4.apache.org,36157,1689771929178 in 201 msec 2023-07-19 13:05:31,576 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-19 13:05:31,577 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=1376f420d426ab5ccf6d1849d6a24f38, ASSIGN in 371 msec 2023-07-19 13:05:31,578 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-19 13:05:31,578 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689771931578"}]},"ts":"1689771931578"} 2023-07-19 13:05:31,581 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-19 13:05:31,585 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-19 13:05:31,588 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 555 msec 2023-07-19 13:05:31,642 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-19 13:05:31,644 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-19 13:05:31,644 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:31,684 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-19 13:05:31,702 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-19 13:05:31,709 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 36 msec 2023-07-19 13:05:31,717 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-19 13:05:31,730 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-19 13:05:31,736 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-07-19 13:05:31,745 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-19 13:05:31,748 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-19 13:05:31,748 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.516sec 2023-07-19 13:05:31,751 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-19 13:05:31,752 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-19 13:05:31,752 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-19 13:05:31,754 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40839,1689771927809-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-19 13:05:31,754 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40839,1689771927809-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-19 13:05:31,764 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-19 13:05:31,805 DEBUG [Listener at localhost/39077] zookeeper.ReadOnlyZKClient(139): Connect 0x53e20c54 to 127.0.0.1:58251 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 13:05:31,810 DEBUG [Listener at localhost/39077] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59ab2568, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 13:05:31,825 DEBUG [hconnection-0x7870815a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 13:05:31,841 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47976, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 13:05:31,850 INFO [Listener at localhost/39077] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:31,861 DEBUG [Listener at localhost/39077] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-19 13:05:31,864 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-19 13:05:31,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-19 13:05:31,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-19 13:05:31,886 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-19 13:05:31,888 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-19 13:05:31,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-19 13:05:31,895 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:31,896 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1 empty. 2023-07-19 13:05:31,898 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:31,898 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-19 13:05:31,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 13:05:31,922 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-19 13:05:31,924 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => cc3a0d3207f00dbe2eb14b149f0091d1, NAME => 't,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/.tmp 2023-07-19 13:05:31,944 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:31,945 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing cc3a0d3207f00dbe2eb14b149f0091d1, disabling compactions & flushes 2023-07-19 13:05:31,945 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:31,945 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:31,945 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. after waiting 0 ms 2023-07-19 13:05:31,945 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:31,945 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:31,945 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for cc3a0d3207f00dbe2eb14b149f0091d1: 2023-07-19 13:05:31,951 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-19 13:05:31,953 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689771931952"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689771931952"}]},"ts":"1689771931952"} 2023-07-19 13:05:31,955 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-19 13:05:31,957 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-19 13:05:31,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689771931957"}]},"ts":"1689771931957"} 2023-07-19 13:05:31,959 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-19 13:05:31,964 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-19 13:05:31,966 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-19 13:05:31,966 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-19 13:05:31,966 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-19 13:05:31,966 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN}] 2023-07-19 13:05:31,969 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN 2023-07-19 13:05:31,971 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39017,1689771929131; forceNewPlan=false, retain=false 2023-07-19 13:05:32,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 13:05:32,121 INFO [jenkins-hbase4:40839] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-19 13:05:32,122 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cc3a0d3207f00dbe2eb14b149f0091d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:32,123 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689771932122"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689771932122"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689771932122"}]},"ts":"1689771932122"} 2023-07-19 13:05:32,126 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure cc3a0d3207f00dbe2eb14b149f0091d1, server=jenkins-hbase4.apache.org,39017,1689771929131}] 2023-07-19 13:05:32,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 13:05:32,280 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:32,281 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 13:05:32,284 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 13:05:32,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cc3a0d3207f00dbe2eb14b149f0091d1, NAME => 't,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:32,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:32,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:32,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:32,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:32,293 INFO [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:32,295 DEBUG [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/f 2023-07-19 13:05:32,295 DEBUG [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/f 2023-07-19 13:05:32,296 INFO [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc3a0d3207f00dbe2eb14b149f0091d1 columnFamilyName f 2023-07-19 13:05:32,298 INFO [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] regionserver.HStore(310): Store=cc3a0d3207f00dbe2eb14b149f0091d1/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:32,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:32,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:32,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:32,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-19 13:05:32,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cc3a0d3207f00dbe2eb14b149f0091d1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64436346, jitterRate=-0.03982362151145935}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 13:05:32,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cc3a0d3207f00dbe2eb14b149f0091d1: 2023-07-19 13:05:32,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1., pid=11, masterSystemTime=1689771932280 2023-07-19 13:05:32,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,321 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cc3a0d3207f00dbe2eb14b149f0091d1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:32,321 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689771932321"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689771932321"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689771932321"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689771932321"}]},"ts":"1689771932321"} 2023-07-19 13:05:32,328 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-19 13:05:32,328 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure cc3a0d3207f00dbe2eb14b149f0091d1, server=jenkins-hbase4.apache.org,39017,1689771929131 in 198 msec 2023-07-19 13:05:32,331 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-19 13:05:32,331 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN in 362 msec 2023-07-19 13:05:32,332 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-19 13:05:32,333 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689771932332"}]},"ts":"1689771932332"} 2023-07-19 13:05:32,335 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-19 13:05:32,338 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-19 13:05:32,341 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 457 msec 2023-07-19 13:05:32,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-19 13:05:32,513 INFO [Listener at localhost/39077] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-19 13:05:32,513 DEBUG [Listener at localhost/39077] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-19 13:05:32,515 INFO [Listener at localhost/39077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-19 13:05:32,522 INFO [Listener at localhost/39077] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-19 13:05:32,523 INFO [Listener at localhost/39077] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-19 13:05:32,523 INFO [Listener at localhost/39077] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-19 13:05:32,527 DEBUG [Listener at localhost/39077] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 13:05:32,535 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53832, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 13:05:32,595 INFO [Listener at localhost/39077] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=580, MaxFileDescriptor=60000, SystemLoadAverage=214, ProcessCount=172, AvailableMemoryMB=6940 2023-07-19 13:05:32,597 INFO [Listener at localhost/39077] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-19 13:05:32,602 DEBUG [Listener at localhost/39077] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 13:05:32,603 DEBUG [Listener at localhost/39077] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,603 INFO [Listener at localhost/39077] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-19 13:05:32,603 INFO [Listener at localhost/39077] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-19 13:05:32,607 INFO [Listener at localhost/39077] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689771931578/Put/vlen=2/seqid=0} 2023-07-19 13:05:32,608 INFO [Listener at localhost/39077] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-19 13:05:32,608 INFO [Listener at localhost/39077] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38./info:regioninfo/1689771931561/Put/vlen=41/seqid=0, hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38./info:seqnumDuringOpen/1689771931561/Put/vlen=8/seqid=0, hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38./info:server/1689771931561/Put/vlen=31/seqid=0, hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38./info:serverstartcode/1689771931561/Put/vlen=8/seqid=0, hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38./info:sn/1689771931361/Put/vlen=45/seqid=0, hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38./info:state/1689771931561/Put/vlen=4/seqid=0} 2023-07-19 13:05:32,608 INFO [Listener at localhost/39077] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 1376f420d426ab5ccf6d1849d6a24f38, NAME => 'hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:32,608 INFO [Listener at localhost/39077] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689771932332/Put/vlen=2/seqid=0} 2023-07-19 13:05:32,608 INFO [Listener at localhost/39077] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-19 13:05:32,608 INFO [Listener at localhost/39077] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1./info:regioninfo/1689771932321/Put/vlen=35/seqid=0, t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1./info:seqnumDuringOpen/1689771932321/Put/vlen=8/seqid=0, t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1./info:server/1689771932321/Put/vlen=31/seqid=0, t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1./info:serverstartcode/1689771932321/Put/vlen=8/seqid=0, t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1./info:sn/1689771932122/Put/vlen=45/seqid=0, t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1./info:state/1689771932321/Put/vlen=4/seqid=0} 2023-07-19 13:05:32,609 INFO [Listener at localhost/39077] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => cc3a0d3207f00dbe2eb14b149f0091d1, NAME => 't,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:32,609 DEBUG [Listener at localhost/39077] zookeeper.ReadOnlyZKClient(139): Connect 0x49865f10 to 127.0.0.1:58251 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 13:05:32,617 DEBUG [Listener at localhost/39077] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e6a4777, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 13:05:32,617 INFO [Listener at localhost/39077] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-19 13:05:32,618 INFO [Listener at localhost/39077] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-19 13:05:32,620 DEBUG [hconnection-0x7c11e80b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 13:05:32,623 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47988, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 13:05:32,627 DEBUG [hconnection-0x7c11e80b-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 13:05:32,629 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53834, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 13:05:32,632 INFO [Listener at localhost/39077] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-19 13:05:32,635 ERROR [Listener at localhost/39077] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,39017,1689771929131: die! ***** 2023-07-19 13:05:32,636 ERROR [Listener at localhost/39077] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-19 13:05:32,640 DEBUG [Listener at localhost/39077] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-19 13:05:32,644 DEBUG [Listener at localhost/39077] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-19 13:05:32,644 DEBUG [Listener at localhost/39077] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-19 13:05:32,645 DEBUG [Listener at localhost/39077] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-19 13:05:32,646 INFO [Listener at localhost/39077] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 889192448, "init": 513802240, "max": 2051014656, "used": 395023328 }, "NonHeapMemoryUsage": { "committed": 112943104, "init": 2555904, "max": -1, "used": 110855616 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-19 13:05:32,684 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40839] master.MasterRpcServices(612): jenkins-hbase4.apache.org,39017,1689771929131 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,39017,1689771929131: die! ***** 2023-07-19 13:05:32,690 INFO [Listener at localhost/39077] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39017,1689771929131' ***** 2023-07-19 13:05:32,690 INFO [Listener at localhost/39077] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-19 13:05:32,691 INFO [RS:0;jenkins-hbase4:39017] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 13:05:32,691 INFO [RS:0;jenkins-hbase4:39017] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-19 13:05:32,691 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 13:05:32,691 INFO [RS:0;jenkins-hbase4:39017] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-19 13:05:32,691 INFO [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(3305): Received CLOSE for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:32,692 INFO [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:32,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39017] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 7101637980514680833 2023-07-19 13:05:32,693 DEBUG [RS:0;jenkins-hbase4:39017] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0725691c to 127.0.0.1:58251 2023-07-19 13:05:32,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cc3a0d3207f00dbe2eb14b149f0091d1, disabling compactions & flushes 2023-07-19 13:05:32,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39017] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39017,1689771929131 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 13:05:32,693 DEBUG [RS:0;jenkins-hbase4:39017] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:32,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39017] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:53834 deadline: 1689771992692, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39017,1689771929131 aborting 2023-07-19 13:05:32,693 INFO [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-19 13:05:32,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,694 DEBUG [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1478): Online Regions={cc3a0d3207f00dbe2eb14b149f0091d1=t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.} 2023-07-19 13:05:32,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. after waiting 0 ms 2023-07-19 13:05:32,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,694 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cc3a0d3207f00dbe2eb14b149f0091d1: 2023-07-19 13:05:32,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:32,695 DEBUG [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1504): Waiting on 2023-07-19 13:05:32,713 DEBUG [Listener at localhost/39077] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39017,1689771929131 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39017,1689771929131 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-19 13:05:32,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39017] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39017,1689771929131 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 13:05:32,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39017] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:53834 deadline: 1689771992716, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39017,1689771929131 aborting 2023-07-19 13:05:32,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39017] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39017,1689771929131 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 13:05:32,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39017] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:53834 deadline: 1689771992825, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39017,1689771929131 aborting 2023-07-19 13:05:32,896 INFO [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,39017,1689771929131; all regions closed. 2023-07-19 13:05:32,905 DEBUG [RS:0;jenkins-hbase4:39017] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:32,906 INFO [RS:0;jenkins-hbase4:39017] regionserver.LeaseManager(133): Closed leases 2023-07-19 13:05:32,906 INFO [RS:0;jenkins-hbase4:39017] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-19 13:05:32,906 INFO [RS:0;jenkins-hbase4:39017] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 13:05:32,906 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 13:05:32,906 INFO [RS:0;jenkins-hbase4:39017] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 13:05:32,907 INFO [RS:0;jenkins-hbase4:39017] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 13:05:32,907 INFO [RS:0;jenkins-hbase4:39017] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39017 2023-07-19 13:05:32,918 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:32,918 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:32,918 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:32,918 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:32,918 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:32,920 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39017,1689771929131] 2023-07-19 13:05:32,920 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39017,1689771929131; numProcessing=1 2023-07-19 13:05:32,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:32,921 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,39017,1689771929131 znode expired, triggering replicatorRemoved event 2023-07-19 13:05:32,922 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39017,1689771929131 already deleted, retry=false 2023-07-19 13:05:32,922 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,39017,1689771929131 on jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:32,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:32,930 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,39017,1689771929131, splitWal=true, meta=false 2023-07-19 13:05:32,930 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,39017,1689771929131 (carryingMeta=false) jenkins-hbase4.apache.org,39017,1689771929131/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@63c38053[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-19 13:05:32,933 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39017,1689771929131, splitWal=true, meta=false 2023-07-19 13:05:32,934 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,39017,1689771929131 had 1 regions 2023-07-19 13:05:32,936 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39017,1689771929131, splitWal=true, meta=false, isMeta: false 2023-07-19 13:05:32,938 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting 2023-07-19 13:05:32,940 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,39017,1689771929131 WAL count=1, meta=false 2023-07-19 13:05:32,943 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530}] 2023-07-19 13:05:32,951 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:32,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530, worker=jenkins-hbase4.apache.org,36157,1689771929178}] 2023-07-19 13:05:33,020 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 13:05:33,021 INFO [RS:0;jenkins-hbase4:39017] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39017,1689771929131; zookeeper connection closed. 2023-07-19 13:05:33,021 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:39017-0x1017dafe8aa0001, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 13:05:33,021 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@35b58976] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@35b58976 2023-07-19 13:05:33,033 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39017 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39017 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 13:05:33,035 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39017 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39017 2023-07-19 13:05:33,117 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36157] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-19 13:05:33,144 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530, size=1.1 K (1148bytes) 2023-07-19 13:05:33,145 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530 2023-07-19 13:05:33,149 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530 after 4ms 2023-07-19 13:05:33,155 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530 took 13ms 2023-07-19 13:05:33,171 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for cc3a0d3207f00dbe2eb14b149f0091d1: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-19 13:05:33,173 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530 so closing down 2023-07-19 13:05:33,173 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 13:05:33,179 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-19 13:05:33,181 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530.temp 2023-07-19 13:05:33,183 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 13:05:33,194 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-19 13:05:33,196 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530.temp to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000013 2023-07-19 13:05:33,196 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 40 ms; skipped=1; WAL=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-19 13:05:33,197 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530, journal: Splitting hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530, size=1.1 K (1148bytes) at 1689771933144Finishing writing output for hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530 so closing down at 1689771933173 (+29 ms)Creating recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530.temp at 1689771933182 (+9 ms)3 split writer threads finished at 1689771933183 (+1 ms)Closed recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689771933194 (+11 ms)Rename recovered edits hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530.temp to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000013 at 1689771933196 (+2 ms)Processed 11 edits across 1 Regions in 40 ms; skipped=1; WAL=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689771933197 (+1 ms) 2023-07-19 13:05:33,197 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530 2023-07-19 13:05:33,200 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-19 13:05:33,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] master.HMaster(3641): Remote procedure done, pid=14 2023-07-19 13:05:33,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] wal.WALSplitUtil(141): Moved hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting/jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530 to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs 2023-07-19 13:05:33,213 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-19 13:05:33,213 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530, worker=jenkins-hbase4.apache.org,36157,1689771929178 in 257 msec 2023-07-19 13:05:33,215 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:33,218 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-19 13:05:33,219 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C39017%2C1689771929131.1689771930530, worker=jenkins-hbase4.apache.org,36157,1689771929178 in 274 msec 2023-07-19 13:05:33,220 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting dir is empty, no logs to split. 2023-07-19 13:05:33,221 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,39017,1689771929131 WAL count=0, meta=false 2023-07-19 13:05:33,221 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39017,1689771929131 WAL splitting is done? wals=0, meta=false 2023-07-19 13:05:33,224 INFO [PEWorker-4] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,39017,1689771929131 failed, ignore...File hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,39017,1689771929131-splitting does not exist. 2023-07-19 13:05:33,225 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN}] 2023-07-19 13:05:33,227 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN 2023-07-19 13:05:33,228 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 13:05:33,341 DEBUG [hconnection-0x7c11e80b-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39017 this server is in the failed servers list 2023-07-19 13:05:33,370 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 13:05:33,378 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=cc3a0d3207f00dbe2eb14b149f0091d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:33,379 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689771933378"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689771933378"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689771933378"}]},"ts":"1689771933378"} 2023-07-19 13:05:33,381 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure cc3a0d3207f00dbe2eb14b149f0091d1, server=jenkins-hbase4.apache.org,36157,1689771929178}] 2023-07-19 13:05:33,540 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:33,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cc3a0d3207f00dbe2eb14b149f0091d1, NAME => 't,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:33,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:33,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:33,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:33,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:33,543 INFO [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:33,545 DEBUG [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/f 2023-07-19 13:05:33,545 DEBUG [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/f 2023-07-19 13:05:33,545 INFO [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc3a0d3207f00dbe2eb14b149f0091d1 columnFamilyName f 2023-07-19 13:05:33,546 INFO [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] regionserver.HStore(310): Store=cc3a0d3207f00dbe2eb14b149f0091d1/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:33,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:33,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:33,553 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000013 2023-07-19 13:05:33,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000013 2023-07-19 13:05:33,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing cc3a0d3207f00dbe2eb14b149f0091d1 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-19 13:05:33,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/.tmp/f/c11601b0ecce4467ab172cdcc5b2e080 2023-07-19 13:05:33,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/.tmp/f/c11601b0ecce4467ab172cdcc5b2e080 as hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/f/c11601b0ecce4467ab172cdcc5b2e080 2023-07-19 13:05:33,670 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/f/c11601b0ecce4467ab172cdcc5b2e080, entries=10, sequenceid=13, filesize=4.8 K 2023-07-19 13:05:33,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for cc3a0d3207f00dbe2eb14b149f0091d1 in 110ms, sequenceid=13, compaction requested=false; wal=null 2023-07-19 13:05:33,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-19 13:05:33,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/0000000000000000013 2023-07-19 13:05:33,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:33,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-19 13:05:33,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cc3a0d3207f00dbe2eb14b149f0091d1; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68214218, jitterRate=0.016471058130264282}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 13:05:33,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cc3a0d3207f00dbe2eb14b149f0091d1: 2023-07-19 13:05:33,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1., pid=16, masterSystemTime=1689771933534 2023-07-19 13:05:33,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:33,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:33,691 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=cc3a0d3207f00dbe2eb14b149f0091d1, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:33,691 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689771933691"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689771933691"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689771933691"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689771933691"}]},"ts":"1689771933691"} 2023-07-19 13:05:33,696 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-19 13:05:33,696 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure cc3a0d3207f00dbe2eb14b149f0091d1, server=jenkins-hbase4.apache.org,36157,1689771929178 in 312 msec 2023-07-19 13:05:33,698 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-19 13:05:33,698 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,39017,1689771929131 after splitting done 2023-07-19 13:05:33,698 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN in 471 msec 2023-07-19 13:05:33,698 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,39017,1689771929131 from processing; numProcessing=0 2023-07-19 13:05:33,701 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,39017,1689771929131, splitWal=true, meta=false in 775 msec 2023-07-19 13:05:33,868 DEBUG [Listener at localhost/39077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49865f10 to 127.0.0.1:58251 2023-07-19 13:05:33,869 DEBUG [Listener at localhost/39077] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:33,869 INFO [Listener at localhost/39077] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-19 13:05:33,886 INFO [Listener at localhost/39077] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=271 (was 294), OpenFileDescriptor=564 (was 580), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 214), ProcessCount=172 (was 172), AvailableMemoryMB=6815 (was 6940) 2023-07-19 13:05:33,902 INFO [Listener at localhost/39077] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=271, OpenFileDescriptor=564, MaxFileDescriptor=60000, SystemLoadAverage=204, ProcessCount=172, AvailableMemoryMB=6814 2023-07-19 13:05:33,903 INFO [Listener at localhost/39077] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:39017 2023-07-19 13:05:33,921 INFO [Listener at localhost/39077] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-19 13:05:33,922 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:33,922 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:33,922 INFO [Listener at localhost/39077] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-19 13:05:33,922 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-19 13:05:33,923 INFO [Listener at localhost/39077] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-19 13:05:33,923 INFO [Listener at localhost/39077] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-19 13:05:33,926 INFO [Listener at localhost/39077] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45863 2023-07-19 13:05:33,926 INFO [Listener at localhost/39077] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-19 13:05:33,928 DEBUG [Listener at localhost/39077] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-19 13:05:33,929 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:33,931 INFO [Listener at localhost/39077] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-19 13:05:33,932 INFO [Listener at localhost/39077] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45863 connecting to ZooKeeper ensemble=127.0.0.1:58251 2023-07-19 13:05:33,936 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:458630x0, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-19 13:05:33,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:45863-0x1017dafe8aa0008 connected 2023-07-19 13:05:33,938 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(162): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-19 13:05:33,939 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(162): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-19 13:05:33,940 DEBUG [Listener at localhost/39077] zookeeper.ZKUtil(164): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-19 13:05:33,942 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45863 2023-07-19 13:05:33,943 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45863 2023-07-19 13:05:33,943 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45863 2023-07-19 13:05:33,946 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45863 2023-07-19 13:05:33,946 DEBUG [Listener at localhost/39077] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45863 2023-07-19 13:05:33,953 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(951): ClusterId : 528e9326-871c-4f3c-b648-fadc18694914 2023-07-19 13:05:33,953 DEBUG [RS:2;jenkins-hbase4:45863] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-19 13:05:33,962 DEBUG [RS:2;jenkins-hbase4:45863] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-19 13:05:33,962 DEBUG [RS:2;jenkins-hbase4:45863] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-19 13:05:33,964 DEBUG [RS:2;jenkins-hbase4:45863] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-19 13:05:33,965 DEBUG [RS:2;jenkins-hbase4:45863] zookeeper.ReadOnlyZKClient(139): Connect 0x285cf0ec to 127.0.0.1:58251 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 13:05:33,975 DEBUG [RS:2;jenkins-hbase4:45863] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@269b0b97, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 13:05:33,975 DEBUG [RS:2;jenkins-hbase4:45863] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22a49b9c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 13:05:33,988 DEBUG [RS:2;jenkins-hbase4:45863] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:45863 2023-07-19 13:05:33,988 INFO [RS:2;jenkins-hbase4:45863] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-19 13:05:33,988 INFO [RS:2;jenkins-hbase4:45863] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-19 13:05:33,988 DEBUG [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1022): About to register with Master. 2023-07-19 13:05:33,995 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,40839,1689771927809 with isa=jenkins-hbase4.apache.org/172.31.14.131:45863, startcode=1689771933921 2023-07-19 13:05:33,995 DEBUG [RS:2;jenkins-hbase4:45863] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-19 13:05:33,999 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47549, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-19 13:05:34,000 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40839] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:34,000 DEBUG [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5 2023-07-19 13:05:34,001 DEBUG [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:33275 2023-07-19 13:05:34,001 DEBUG [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-19 13:05:34,004 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:34,007 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:34,008 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,45863,1689771933921] 2023-07-19 13:05:34,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:34,013 DEBUG [RS:2;jenkins-hbase4:45863] zookeeper.ZKUtil(162): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:34,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:34,013 WARN [RS:2;jenkins-hbase4:45863] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-19 13:05:34,013 INFO [RS:2;jenkins-hbase4:45863] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 13:05:34,013 DEBUG [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1948): logDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:34,019 DEBUG [RS:2;jenkins-hbase4:45863] zookeeper.ZKUtil(162): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:34,020 DEBUG [RS:2;jenkins-hbase4:45863] zookeeper.ZKUtil(162): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:34,022 DEBUG [RS:2;jenkins-hbase4:45863] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-19 13:05:34,022 INFO [RS:2;jenkins-hbase4:45863] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-19 13:05:34,024 INFO [RS:2;jenkins-hbase4:45863] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-19 13:05:34,026 INFO [RS:2;jenkins-hbase4:45863] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-19 13:05:34,026 INFO [RS:2;jenkins-hbase4:45863] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:34,026 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-19 13:05:34,028 INFO [RS:2;jenkins-hbase4:45863] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:34,028 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:34,028 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:34,029 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:34,029 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:34,029 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:34,029 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-19 13:05:34,029 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:34,029 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:34,029 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:34,029 DEBUG [RS:2;jenkins-hbase4:45863] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-19 13:05:34,031 INFO [RS:2;jenkins-hbase4:45863] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:34,031 INFO [RS:2;jenkins-hbase4:45863] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:34,031 INFO [RS:2;jenkins-hbase4:45863] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:34,043 INFO [RS:2;jenkins-hbase4:45863] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-19 13:05:34,043 INFO [RS:2;jenkins-hbase4:45863] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45863,1689771933921-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-19 13:05:34,058 INFO [RS:2;jenkins-hbase4:45863] regionserver.Replication(203): jenkins-hbase4.apache.org,45863,1689771933921 started 2023-07-19 13:05:34,059 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,45863,1689771933921, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:45863, sessionid=0x1017dafe8aa0008 2023-07-19 13:05:34,059 INFO [Listener at localhost/39077] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:45863,5,FailOnTimeoutGroup] 2023-07-19 13:05:34,059 DEBUG [RS:2;jenkins-hbase4:45863] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-19 13:05:34,059 INFO [Listener at localhost/39077] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:34,059 DEBUG [RS:2;jenkins-hbase4:45863] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:34,059 INFO [Listener at localhost/39077] client.TestScannerTimeout(177): START ************ test3686b 2023-07-19 13:05:34,059 DEBUG [RS:2;jenkins-hbase4:45863] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45863,1689771933921' 2023-07-19 13:05:34,059 DEBUG [RS:2;jenkins-hbase4:45863] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-19 13:05:34,060 DEBUG [RS:2;jenkins-hbase4:45863] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-19 13:05:34,060 DEBUG [RS:2;jenkins-hbase4:45863] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-19 13:05:34,060 DEBUG [RS:2;jenkins-hbase4:45863] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-19 13:05:34,061 DEBUG [RS:2;jenkins-hbase4:45863] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:34,061 DEBUG [RS:2;jenkins-hbase4:45863] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45863,1689771933921' 2023-07-19 13:05:34,061 DEBUG [RS:2;jenkins-hbase4:45863] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-19 13:05:34,061 DEBUG [RS:2;jenkins-hbase4:45863] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-19 13:05:34,062 DEBUG [RS:2;jenkins-hbase4:45863] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-19 13:05:34,062 INFO [RS:2;jenkins-hbase4:45863] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-19 13:05:34,062 INFO [RS:2;jenkins-hbase4:45863] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-19 13:05:34,063 DEBUG [Listener at localhost/39077] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 13:05:34,063 DEBUG [Listener at localhost/39077] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:34,064 DEBUG [Listener at localhost/39077] zookeeper.ReadOnlyZKClient(139): Connect 0x6e13ade1 to 127.0.0.1:58251 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 13:05:34,073 DEBUG [Listener at localhost/39077] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18d71e63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 13:05:34,076 DEBUG [hconnection-0x5ac48154-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 13:05:34,079 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48002, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 13:05:34,165 INFO [RS:2;jenkins-hbase4:45863] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45863%2C1689771933921, suffix=, logDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,45863,1689771933921, archiveDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs, maxLogs=32 2023-07-19 13:05:34,186 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK] 2023-07-19 13:05:34,186 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK] 2023-07-19 13:05:34,189 INFO [RS:2;jenkins-hbase4:45863] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,45863,1689771933921/jenkins-hbase4.apache.org%2C45863%2C1689771933921.1689771934166 2023-07-19 13:05:34,190 DEBUG [RS:2;jenkins-hbase4:45863] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK], DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK]] 2023-07-19 13:05:36,300 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-19 13:05:36,349 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-19 13:05:36,350 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-19 13:05:37,999 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 13:05:38,000 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-19 13:05:49,085 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease -1598724721791205357 expired clientIPAndPort=172.31.14.131:48002, userName=jenkins, regionInfo=t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:51,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36157] regionserver.RSRpcServices(3094): Client tried to access missing scanner -1598724721791205357 2023-07-19 13:05:51,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36157] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:48002 deadline: 1689772011087, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-1598724721791205357'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-19 13:05:51,089 DEBUG [Listener at localhost/39077] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-1598724721791205357'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-1598724721791205357'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-1598724721791205357'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-19 13:05:51,094 DEBUG [Listener at localhost/39077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e13ade1 to 127.0.0.1:58251 2023-07-19 13:05:51,094 DEBUG [Listener at localhost/39077] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:51,094 INFO [Listener at localhost/39077] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-19 13:05:51,114 INFO [Listener at localhost/39077] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=294 (was 271) Potentially hanging thread: PacketResponder: BP-1815553540-172.31.14.131-1689771924702:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58251@0x285cf0ec-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:33275 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-92694992_17 at /127.0.0.1:46034 [Receiving block BP-1815553540-172.31.14.131-1689771924702:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-92694992_17 at /127.0.0.1:36208 [Receiving block BP-1815553540-172.31.14.131-1689771924702:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase4:45863Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS:2;jenkins-hbase4:45863 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5-prefix:jenkins-hbase4.apache.org,45863,1689771933921 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58251@0x6e13ade1 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1696317946.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58251@0x285cf0ec sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1696317946.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58251@0x6e13ade1-SendThread(127.0.0.1:58251) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58251@0x6e13ade1-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58251@0x285cf0ec-SendThread(127.0.0.1:58251) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: PacketResponder: BP-1815553540-172.31.14.131-1689771924702:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45863 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:2;jenkins-hbase4:45863-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=571 (was 564) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 204), ProcessCount=172 (was 172), AvailableMemoryMB=6619 (was 6814) 2023-07-19 13:05:51,132 INFO [Listener at localhost/39077] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=294, OpenFileDescriptor=571, MaxFileDescriptor=60000, SystemLoadAverage=204, ProcessCount=172, AvailableMemoryMB=6615 2023-07-19 13:05:51,132 INFO [Listener at localhost/39077] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:39017 2023-07-19 13:05:51,132 INFO [Listener at localhost/39077] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,39017,1689771929131 2023-07-19 13:05:51,133 INFO [Listener at localhost/39077] client.TestScannerTimeout(104): START************ test2772 2023-07-19 13:05:51,137 DEBUG [Listener at localhost/39077] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-19 13:05:51,137 DEBUG [Listener at localhost/39077] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:51,138 DEBUG [Listener at localhost/39077] zookeeper.ReadOnlyZKClient(139): Connect 0x7cfe3418 to 127.0.0.1:58251 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-19 13:05:51,143 DEBUG [Listener at localhost/39077] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2376f577, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-19 13:05:51,144 ERROR [Listener at localhost/39077] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,36157,1689771929178: die! ***** 2023-07-19 13:05:51,144 ERROR [Listener at localhost/39077] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-19 13:05:51,144 DEBUG [Listener at localhost/39077] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-19 13:05:51,145 DEBUG [Listener at localhost/39077] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-19 13:05:51,154 DEBUG [Listener at localhost/39077] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-19 13:05:51,157 DEBUG [Listener at localhost/39077] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-19 13:05:51,179 INFO [Listener at localhost/39077] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 984088576, "init": 513802240, "max": 2051014656, "used": 259684488 }, "NonHeapMemoryUsage": { "committed": 122183680, "init": 2555904, "max": -1, "used": 119709032 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11957, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 197, "ResponseSize_max": 679, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 199, "ResponseSize_median": 202, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 678, "ResponseSize_95th_percentile": 678, "ResponseSize_98th_percentile": 678, "ResponseSize_99th_percentile": 678, "ResponseSize_99.9th_percentile": 678, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16683 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58251", "tag.serverName": "jenkins-hbase4.apache.org,36157,1689771929178", "tag.clusterId": "528e9326-871c-4f3c-b648-fadc18694914", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 1, "maxStoreFileCount": 1, "memStoreSize": 3306, "storeFileSize": 4926, "maxStoreFileAge": 16721, "minStoreFileAge": 16721, "avgStoreFileAge": 16721, "numReferenceFiles": 0, "regionServerStartTime": 1689771929178, "averageRegionSize": 2744, "storeFileIndexSize": 320, "staticIndexSize": 31, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 1863680, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 46, "totalRowActionRequestCount": 57, "readRequestCount": 43, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 10, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 22, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 10, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 270, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 21, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 10, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 21, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 270, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.593375734291479, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.1526112566744136, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-19 13:05:51,181 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40839] master.MasterRpcServices(612): jenkins-hbase4.apache.org,36157,1689771929178 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,36157,1689771929178: die! ***** 2023-07-19 13:05:51,190 INFO [Listener at localhost/39077] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,36157,1689771929178' ***** 2023-07-19 13:05:51,192 INFO [Listener at localhost/39077] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-19 13:05:51,192 INFO [RS:1;jenkins-hbase4:36157] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 13:05:51,193 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 13:05:51,193 INFO [RS:1;jenkins-hbase4:36157] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-19 13:05:51,193 INFO [RS:1;jenkins-hbase4:36157] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-19 13:05:51,193 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(3305): Received CLOSE for 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:51,194 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(3305): Received CLOSE for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:51,195 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:51,195 DEBUG [RS:1;jenkins-hbase4:36157] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b40a713 to 127.0.0.1:58251 2023-07-19 13:05:51,195 DEBUG [RS:1;jenkins-hbase4:36157] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:51,195 INFO [RS:1;jenkins-hbase4:36157] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 13:05:51,195 INFO [RS:1;jenkins-hbase4:36157] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 13:05:51,195 INFO [RS:1;jenkins-hbase4:36157] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 13:05:51,195 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-19 13:05:51,196 DEBUG [hconnection-0x5dc4b8b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 13:05:51,197 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-19 13:05:51,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1376f420d426ab5ccf6d1849d6a24f38, disabling compactions & flushes 2023-07-19 13:05:51,197 DEBUG [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1478): Online Regions={1376f420d426ab5ccf6d1849d6a24f38=hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38., 1588230740=hbase:meta,,1.1588230740, cc3a0d3207f00dbe2eb14b149f0091d1=t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.} 2023-07-19 13:05:51,197 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 13:05:51,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:51,198 DEBUG [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1504): Waiting on 1376f420d426ab5ccf6d1849d6a24f38, 1588230740, cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:51,198 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 13:05:51,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:51,198 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 13:05:51,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. after waiting 0 ms 2023-07-19 13:05:51,198 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 13:05:51,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:51,198 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 13:05:51,201 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54580, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 13:05:51,201 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36157] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,36157,1689771929178 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 13:05:51,201 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:51,201 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36157] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:54580 deadline: 1689772011201, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,36157,1689771929178 aborting 2023-07-19 13:05:51,201 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 3228 in region hbase:meta,,1.1588230740 2023-07-19 13:05:51,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:51,202 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 13:05:51,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1376f420d426ab5ccf6d1849d6a24f38: 2023-07-19 13:05:51,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:51,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cc3a0d3207f00dbe2eb14b149f0091d1, disabling compactions & flushes 2023-07-19 13:05:51,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:51,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:51,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. after waiting 0 ms 2023-07-19 13:05:51,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:51,204 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 13:05:51,205 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 13:05:51,205 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-19 13:05:51,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:51,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cc3a0d3207f00dbe2eb14b149f0091d1: 2023-07-19 13:05:51,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:51,228 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 13:05:51,307 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36157] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,36157,1689771929178 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-19 13:05:51,307 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36157] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:54580 deadline: 1689772011306, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,36157,1689771929178 aborting 2023-07-19 13:05:51,372 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-19 13:05:51,372 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-19 13:05:51,398 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,36157,1689771929178; all regions closed. 2023-07-19 13:05:51,412 DEBUG [RS:1;jenkins-hbase4:36157] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:51,412 INFO [RS:1;jenkins-hbase4:36157] regionserver.LeaseManager(133): Closed leases 2023-07-19 13:05:51,412 INFO [RS:1;jenkins-hbase4:36157] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-19 13:05:51,413 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 13:05:51,413 INFO [RS:1;jenkins-hbase4:36157] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36157 2023-07-19 13:05:51,418 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:51,418 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:51,418 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 2023-07-19 13:05:51,418 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:51,418 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:51,419 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36157,1689771929178] 2023-07-19 13:05:51,419 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36157,1689771929178; numProcessing=1 2023-07-19 13:05:51,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:51,427 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,36157,1689771929178 znode expired, triggering replicatorRemoved event 2023-07-19 13:05:51,428 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36157,1689771929178 already deleted, retry=false 2023-07-19 13:05:51,428 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,36157,1689771929178 on jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:51,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:51,429 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,36157,1689771929178, splitWal=true, meta=true 2023-07-19 13:05:51,429 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase4.apache.org,36157,1689771929178 (carryingMeta=true) jenkins-hbase4.apache.org,36157,1689771929178/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@169812b[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-19 13:05:51,433 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,36157,1689771929178, splitWal=true, meta=true 2023-07-19 13:05:51,434 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,36157,1689771929178, splitWal=true, meta=true, isMeta: true 2023-07-19 13:05:51,440 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting 2023-07-19 13:05:51,442 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,36157,1689771929178 WAL count=1, meta=true 2023-07-19 13:05:51,442 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta}] 2023-07-19 13:05:51,444 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:51,445 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta, worker=jenkins-hbase4.apache.org,45863,1689771933921}] 2023-07-19 13:05:51,510 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:36157 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:36157 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 13:05:51,511 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:36157 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:36157 2023-07-19 13:05:51,528 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 13:05:51,528 INFO [RS:1;jenkins-hbase4:36157] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,36157,1689771929178; zookeeper connection closed. 2023-07-19 13:05:51,528 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:36157-0x1017dafe8aa0002, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 13:05:51,528 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7d99d621] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7d99d621 2023-07-19 13:05:51,598 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:51,598 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-19 13:05:51,600 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40864, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-19 13:05:51,600 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45863] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-19 13:05:51,616 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta, size=4.0 K (4123bytes) 2023-07-19 13:05:51,616 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta 2023-07-19 13:05:51,617 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta after 1ms 2023-07-19 13:05:51,622 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta took 6ms 2023-07-19 13:05:51,626 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-19 13:05:51,628 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta so closing down 2023-07-19 13:05:51,628 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 13:05:51,632 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta.temp 2023-07-19 13:05:51,632 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 13:05:51,642 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-19 13:05:51,644 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta.temp to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-19 13:05:51,644 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 22 ms; skipped=1; WAL=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false 2023-07-19 13:05:51,644 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta, journal: Splitting hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta, size=4.0 K (4123bytes) at 1689771951616Finishing writing output for hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta so closing down at 1689771951628 (+12 ms)Creating recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta.temp at 1689771951632 (+4 ms)3 split writer threads finished at 1689771951632Closed recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689771951642 (+10 ms)Rename recovered edits hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta.temp to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689771951644 (+2 ms)Processed 13 edits across 1 Regions in 22 ms; skipped=1; WAL=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false at 1689771951644 2023-07-19 13:05:51,644 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta 2023-07-19 13:05:51,644 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-19 13:05:51,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] master.HMaster(3641): Remote procedure done, pid=19 2023-07-19 13:05:51,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] wal.WALSplitUtil(141): Moved hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs 2023-07-19 13:05:51,651 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-19 13:05:51,652 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta, worker=jenkins-hbase4.apache.org,45863,1689771933921 in 204 msec 2023-07-19 13:05:51,653 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:51,656 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-19 13:05:51,656 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C36157%2C1689771929178.meta.1689771930718.meta, worker=jenkins-hbase4.apache.org,45863,1689771933921 in 211 msec 2023-07-19 13:05:51,658 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting dir is empty, no logs to split. 2023-07-19 13:05:51,658 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,36157,1689771929178 WAL count=0, meta=true 2023-07-19 13:05:51,658 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,36157,1689771929178 WAL splitting is done? wals=0, meta=true 2023-07-19 13:05:51,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-19 13:05:51,661 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-19 13:05:51,661 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 13:05:51,812 DEBUG [jenkins-hbase4:40839] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 13:05:51,812 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45863,1689771933921, state=OPENING 2023-07-19 13:05:51,813 DEBUG [hconnection-0x5dc4b8b8-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:36157 this server is in the failed servers list 2023-07-19 13:05:51,814 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 13:05:51,814 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45863,1689771933921}] 2023-07-19 13:05:51,814 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 13:05:51,973 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-19 13:05:51,973 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-19 13:05:51,975 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45863%2C1689771933921.meta, suffix=.meta, logDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,45863,1689771933921, archiveDir=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs, maxLogs=32 2023-07-19 13:05:51,992 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK] 2023-07-19 13:05:51,993 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK] 2023-07-19 13:05:51,995 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,45863,1689771933921/jenkins-hbase4.apache.org%2C45863%2C1689771933921.meta.1689771951976.meta 2023-07-19 13:05:51,995 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33723,DS-d790c55a-9d91-427b-ad7b-0a1912fa7255,DISK], DatanodeInfoWithStorage[127.0.0.1:35447,DS-1a11a592-8a3b-4a7b-b2c0-d3916e765007,DISK]] 2023-07-19 13:05:51,995 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:51,996 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-19 13:05:51,996 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-19 13:05:51,996 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-19 13:05:51,996 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-19 13:05:51,996 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:51,996 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-19 13:05:51,996 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-19 13:05:52,000 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-19 13:05:52,001 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/info 2023-07-19 13:05:52,001 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/info 2023-07-19 13:05:52,001 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-19 13:05:52,003 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:52,003 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-19 13:05:52,005 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/rep_barrier 2023-07-19 13:05:52,005 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/rep_barrier 2023-07-19 13:05:52,005 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-19 13:05:52,006 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:52,006 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-19 13:05:52,008 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/table 2023-07-19 13:05:52,008 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/table 2023-07-19 13:05:52,008 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-19 13:05:52,009 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:52,010 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740 2023-07-19 13:05:52,012 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740 2023-07-19 13:05:52,013 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-19 13:05:52,022 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-19 13:05:52,022 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.15 KB heapSize=6.27 KB 2023-07-19 13:05:52,040 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/.tmp/info/566be384e0b44df8b5eade7f369feaf3 2023-07-19 13:05:52,063 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/.tmp/table/54f16d42586b4fd5a4d196556e2bf57d 2023-07-19 13:05:52,071 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/.tmp/info/566be384e0b44df8b5eade7f369feaf3 as hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/info/566be384e0b44df8b5eade7f369feaf3 2023-07-19 13:05:52,080 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/info/566be384e0b44df8b5eade7f369feaf3, entries=28, sequenceid=15, filesize=7.7 K 2023-07-19 13:05:52,082 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/.tmp/table/54f16d42586b4fd5a4d196556e2bf57d as hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/table/54f16d42586b4fd5a4d196556e2bf57d 2023-07-19 13:05:52,093 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/table/54f16d42586b4fd5a4d196556e2bf57d, entries=4, sequenceid=15, filesize=4.7 K 2023-07-19 13:05:52,094 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.15 KB/3228, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 72ms, sequenceid=15, compaction requested=false; wal=null 2023-07-19 13:05:52,095 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-19 13:05:52,098 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-19 13:05:52,100 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-19 13:05:52,103 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-19 13:05:52,104 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73040110, jitterRate=0.08838245272636414}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-19 13:05:52,104 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-19 13:05:52,105 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689771951967 2023-07-19 13:05:52,108 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-19 13:05:52,108 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-19 13:05:52,109 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45863,1689771933921, state=OPEN 2023-07-19 13:05:52,111 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-19 13:05:52,111 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-19 13:05:52,113 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-19 13:05:52,113 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45863,1689771933921 in 297 msec 2023-07-19 13:05:52,116 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-19 13:05:52,116 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,36157,1689771929178 had 3 regions 2023-07-19 13:05:52,116 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 454 msec 2023-07-19 13:05:52,117 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,36157,1689771929178, splitWal=true, meta=true, isMeta: false 2023-07-19 13:05:52,119 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,36157,1689771929178 WAL count=1, meta=false 2023-07-19 13:05:52,120 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530}] 2023-07-19 13:05:52,124 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:52,125 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530, worker=jenkins-hbase4.apache.org,45863,1689771933921}] 2023-07-19 13:05:52,279 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45863] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-19 13:05:52,292 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530, size=899 (899bytes) 2023-07-19 13:05:52,292 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530 2023-07-19 13:05:52,293 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530 after 1ms 2023-07-19 13:05:52,298 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530 took 6ms 2023-07-19 13:05:52,301 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 1376f420d426ab5ccf6d1849d6a24f38: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-19 13:05:52,303 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for cc3a0d3207f00dbe2eb14b149f0091d1: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-19 13:05:52,303 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530 so closing down 2023-07-19 13:05:52,304 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-19 13:05:52,306 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530.temp 2023-07-19 13:05:52,306 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-19 13:05:52,317 DEBUG [hconnection-0x5dc4b8b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 13:05:52,324 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-19 13:05:52,325 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40876, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 13:05:52,327 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530.temp to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000005 2023-07-19 13:05:52,327 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 29 ms; skipped=2; WAL=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530, size=899, length=899, corrupted=false, cancelled=false 2023-07-19 13:05:52,327 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530, journal: Splitting hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530, size=899 (899bytes) at 1689771952292Finishing writing output for hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530 so closing down at 1689771952303 (+11 ms)Creating recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530.temp at 1689771952306 (+3 ms)3 split writer threads finished at 1689771952306Closed recovered edits writer path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689771952324 (+18 ms)Rename recovered edits hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530.temp to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000005 at 1689771952327 (+3 ms)Processed 4 edits across 1 Regions in 29 ms; skipped=2; WAL=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530, size=899, length=899, corrupted=false, cancelled=false at 1689771952327 2023-07-19 13:05:52,327 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530 2023-07-19 13:05:52,327 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-19 13:05:52,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] master.HMaster(3641): Remote procedure done, pid=23 2023-07-19 13:05:52,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40839] wal.WALSplitUtil(141): Moved hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting/jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530 to hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs 2023-07-19 13:05:52,334 DEBUG [hconnection-0x5dc4b8b8-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:36157 this server is in the failed servers list 2023-07-19 13:05:52,337 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-19 13:05:52,338 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530, worker=jenkins-hbase4.apache.org,45863,1689771933921 in 210 msec 2023-07-19 13:05:52,339 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:52,341 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-19 13:05:52,341 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C36157%2C1689771929178.1689771930530, worker=jenkins-hbase4.apache.org,45863,1689771933921 in 220 msec 2023-07-19 13:05:52,343 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/WALs/jenkins-hbase4.apache.org,36157,1689771929178-splitting dir is empty, no logs to split. 2023-07-19 13:05:52,343 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,36157,1689771929178 WAL count=0, meta=false 2023-07-19 13:05:52,344 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,36157,1689771929178 WAL splitting is done? wals=0, meta=false 2023-07-19 13:05:52,345 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1376f420d426ab5ccf6d1849d6a24f38, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN}] 2023-07-19 13:05:52,347 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1376f420d426ab5ccf6d1849d6a24f38, ASSIGN 2023-07-19 13:05:52,347 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN 2023-07-19 13:05:52,348 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 13:05:52,348 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=1376f420d426ab5ccf6d1849d6a24f38, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-19 13:05:52,348 DEBUG [jenkins-hbase4:40839] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-19 13:05:52,349 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=cc3a0d3207f00dbe2eb14b149f0091d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:52,349 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=1376f420d426ab5ccf6d1849d6a24f38, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:52,349 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689771952349"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689771952349"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689771952349"}]},"ts":"1689771952349"} 2023-07-19 13:05:52,349 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689771952349"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689771952349"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689771952349"}]},"ts":"1689771952349"} 2023-07-19 13:05:52,350 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:36157 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:36157 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-19 13:05:52,351 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:36157 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:36157 2023-07-19 13:05:52,442 DEBUG [hconnection-0x5dc4b8b8-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:36157 this server is in the failed servers list 2023-07-19 13:05:52,453 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-19 13:05:52,454 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-19 13:05:52,456 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; OpenRegionProcedure cc3a0d3207f00dbe2eb14b149f0091d1, server=jenkins-hbase4.apache.org,45863,1689771933921}] 2023-07-19 13:05:52,457 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure 1376f420d426ab5ccf6d1849d6a24f38, server=jenkins-hbase4.apache.org,45863,1689771933921}] 2023-07-19 13:05:52,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:52,614 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cc3a0d3207f00dbe2eb14b149f0091d1, NAME => 't,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:52,615 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:52,615 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:52,615 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:52,615 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:52,617 INFO [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:52,618 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:52,618 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1376f420d426ab5ccf6d1849d6a24f38, NAME => 'hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.', STARTKEY => '', ENDKEY => ''} 2023-07-19 13:05:52,618 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:52,618 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-19 13:05:52,619 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:52,619 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:52,619 DEBUG [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/f 2023-07-19 13:05:52,619 DEBUG [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/f 2023-07-19 13:05:52,619 INFO [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc3a0d3207f00dbe2eb14b149f0091d1 columnFamilyName f 2023-07-19 13:05:52,626 INFO [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:52,628 DEBUG [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/info 2023-07-19 13:05:52,628 DEBUG [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/info 2023-07-19 13:05:52,628 INFO [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1376f420d426ab5ccf6d1849d6a24f38 columnFamilyName info 2023-07-19 13:05:52,629 INFO [StoreOpener-1376f420d426ab5ccf6d1849d6a24f38-1] regionserver.HStore(310): Store=1376f420d426ab5ccf6d1849d6a24f38/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:52,631 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:52,635 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:52,636 DEBUG [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] regionserver.HStore(539): loaded hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/f/c11601b0ecce4467ab172cdcc5b2e080 2023-07-19 13:05:52,636 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000005 2023-07-19 13:05:52,637 INFO [StoreOpener-cc3a0d3207f00dbe2eb14b149f0091d1-1] regionserver.HStore(310): Store=cc3a0d3207f00dbe2eb14b149f0091d1/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-19 13:05:52,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:52,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:52,641 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000005 2023-07-19 13:05:52,641 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1376f420d426ab5ccf6d1849d6a24f38 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-19 13:05:52,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:52,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-19 13:05:52,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cc3a0d3207f00dbe2eb14b149f0091d1; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68916638, jitterRate=0.026937931776046753}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 13:05:52,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cc3a0d3207f00dbe2eb14b149f0091d1: 2023-07-19 13:05:52,651 DEBUG [hconnection-0x5dc4b8b8-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:36157 this server is in the failed servers list 2023-07-19 13:05:52,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1., pid=26, masterSystemTime=1689771952609 2023-07-19 13:05:52,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:52,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:52,657 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=cc3a0d3207f00dbe2eb14b149f0091d1, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:52,657 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689771952657"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689771952657"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689771952657"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689771952657"}]},"ts":"1689771952657"} 2023-07-19 13:05:52,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-19 13:05:52,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; OpenRegionProcedure cc3a0d3207f00dbe2eb14b149f0091d1, server=jenkins-hbase4.apache.org,45863,1689771933921 in 203 msec 2023-07-19 13:05:52,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=cc3a0d3207f00dbe2eb14b149f0091d1, ASSIGN in 317 msec 2023-07-19 13:05:52,679 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/.tmp/info/c22d606b11c540769b8040bfe243408e 2023-07-19 13:05:52,691 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/.tmp/info/c22d606b11c540769b8040bfe243408e as hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/info/c22d606b11c540769b8040bfe243408e 2023-07-19 13:05:52,700 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/info/c22d606b11c540769b8040bfe243408e, entries=2, sequenceid=5, filesize=4.8 K 2023-07-19 13:05:52,701 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 1376f420d426ab5ccf6d1849d6a24f38 in 60ms, sequenceid=5, compaction requested=false; wal=null 2023-07-19 13:05:52,702 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/0000000000000000005 2023-07-19 13:05:52,707 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:52,710 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-19 13:05:52,711 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1376f420d426ab5ccf6d1849d6a24f38; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63974846, jitterRate=-0.046700507402420044}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-19 13:05:52,712 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1376f420d426ab5ccf6d1849d6a24f38: 2023-07-19 13:05:52,713 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38., pid=27, masterSystemTime=1689771952609 2023-07-19 13:05:52,715 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:52,715 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:52,716 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=1376f420d426ab5ccf6d1849d6a24f38, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:52,716 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689771952716"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689771952716"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689771952716"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689771952716"}]},"ts":"1689771952716"} 2023-07-19 13:05:52,721 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-07-19 13:05:52,721 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure 1376f420d426ab5ccf6d1849d6a24f38, server=jenkins-hbase4.apache.org,45863,1689771933921 in 261 msec 2023-07-19 13:05:52,723 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=17 2023-07-19 13:05:52,724 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,36157,1689771929178 after splitting done 2023-07-19 13:05:52,724 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=1376f420d426ab5ccf6d1849d6a24f38, ASSIGN in 376 msec 2023-07-19 13:05:52,724 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase4.apache.org,36157,1689771929178 from processing; numProcessing=0 2023-07-19 13:05:52,726 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,36157,1689771929178, splitWal=true, meta=true in 1.2960 sec 2023-07-19 13:05:52,963 DEBUG [Listener at localhost/39077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7cfe3418 to 127.0.0.1:58251 2023-07-19 13:05:52,963 DEBUG [Listener at localhost/39077] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:52,963 INFO [Listener at localhost/39077] client.TestScannerTimeout(124): END ************ test2772 2023-07-19 13:05:52,978 INFO [Listener at localhost/39077] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=276 (was 294), OpenFileDescriptor=580 (was 571) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 204), ProcessCount=172 (was 172), AvailableMemoryMB=6541 (was 6615) 2023-07-19 13:05:52,978 INFO [Listener at localhost/39077] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-19 13:05:52,978 INFO [Listener at localhost/39077] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-19 13:05:52,978 DEBUG [Listener at localhost/39077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53e20c54 to 127.0.0.1:58251 2023-07-19 13:05:52,979 DEBUG [Listener at localhost/39077] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:52,979 DEBUG [Listener at localhost/39077] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-19 13:05:52,979 DEBUG [Listener at localhost/39077] util.JVMClusterUtil(257): Found active master hash=2100485527, stopped=false 2023-07-19 13:05:52,979 INFO [Listener at localhost/39077] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:52,985 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-19 13:05:52,985 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-19 13:05:52,985 INFO [Listener at localhost/39077] procedure2.ProcedureExecutor(629): Stopping 2023-07-19 13:05:52,985 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:52,986 DEBUG [Listener at localhost/39077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x788474c0 to 127.0.0.1:58251 2023-07-19 13:05:52,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 13:05:52,986 DEBUG [Listener at localhost/39077] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:52,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-19 13:05:52,987 INFO [Listener at localhost/39077] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,45863,1689771933921' ***** 2023-07-19 13:05:52,987 INFO [Listener at localhost/39077] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-19 13:05:52,987 INFO [RS:2;jenkins-hbase4:45863] regionserver.HeapMemoryManager(220): Stopping 2023-07-19 13:05:52,987 INFO [RS:2;jenkins-hbase4:45863] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-19 13:05:52,987 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-19 13:05:52,987 INFO [RS:2;jenkins-hbase4:45863] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-19 13:05:52,987 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(3305): Received CLOSE for 1376f420d426ab5ccf6d1849d6a24f38 2023-07-19 13:05:52,988 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(3305): Received CLOSE for cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:52,988 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:52,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1376f420d426ab5ccf6d1849d6a24f38, disabling compactions & flushes 2023-07-19 13:05:52,988 DEBUG [RS:2;jenkins-hbase4:45863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x285cf0ec to 127.0.0.1:58251 2023-07-19 13:05:52,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:52,988 DEBUG [RS:2;jenkins-hbase4:45863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:52,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:52,989 INFO [RS:2;jenkins-hbase4:45863] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-19 13:05:52,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. after waiting 0 ms 2023-07-19 13:05:52,989 INFO [RS:2;jenkins-hbase4:45863] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-19 13:05:52,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:52,989 INFO [RS:2;jenkins-hbase4:45863] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-19 13:05:52,989 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-19 13:05:52,989 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-19 13:05:52,989 DEBUG [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1478): Online Regions={1376f420d426ab5ccf6d1849d6a24f38=hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38., 1588230740=hbase:meta,,1.1588230740, cc3a0d3207f00dbe2eb14b149f0091d1=t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1.} 2023-07-19 13:05:52,990 DEBUG [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1504): Waiting on 1376f420d426ab5ccf6d1849d6a24f38, 1588230740, cc3a0d3207f00dbe2eb14b149f0091d1 2023-07-19 13:05:52,990 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-19 13:05:52,991 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-19 13:05:52,991 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-19 13:05:52,991 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-19 13:05:52,991 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-19 13:05:52,991 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-19 13:05:52,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/namespace/1376f420d426ab5ccf6d1849d6a24f38/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-19 13:05:52,997 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:52,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1376f420d426ab5ccf6d1849d6a24f38: 2023-07-19 13:05:52,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689771931028.1376f420d426ab5ccf6d1849d6a24f38. 2023-07-19 13:05:52,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cc3a0d3207f00dbe2eb14b149f0091d1, disabling compactions & flushes 2023-07-19 13:05:52,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:52,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:52,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. after waiting 0 ms 2023-07-19 13:05:52,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:53,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/default/t/cc3a0d3207f00dbe2eb14b149f0091d1/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-19 13:05:53,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:53,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cc3a0d3207f00dbe2eb14b149f0091d1: 2023-07-19 13:05:53,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689771931877.cc3a0d3207f00dbe2eb14b149f0091d1. 2023-07-19 13:05:53,019 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/.tmp/info/97b52298dae247e1a41fa1622ef49172 2023-07-19 13:05:53,028 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/.tmp/info/97b52298dae247e1a41fa1622ef49172 as hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/info/97b52298dae247e1a41fa1622ef49172 2023-07-19 13:05:53,031 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-19 13:05:53,031 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-19 13:05:53,033 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-19 13:05:53,036 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/info/97b52298dae247e1a41fa1622ef49172, entries=16, sequenceid=22, filesize=6.4 K 2023-07-19 13:05:53,037 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 46ms, sequenceid=22, compaction requested=false 2023-07-19 13:05:53,044 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-19 13:05:53,045 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-19 13:05:53,046 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-19 13:05:53,046 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-19 13:05:53,046 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-19 13:05:53,190 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,45863,1689771933921; all regions closed. 2023-07-19 13:05:53,197 DEBUG [RS:2;jenkins-hbase4:45863] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs 2023-07-19 13:05:53,197 INFO [RS:2;jenkins-hbase4:45863] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C45863%2C1689771933921.meta:.meta(num 1689771951976) 2023-07-19 13:05:53,202 DEBUG [RS:2;jenkins-hbase4:45863] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/oldWALs 2023-07-19 13:05:53,203 INFO [RS:2;jenkins-hbase4:45863] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C45863%2C1689771933921:(num 1689771934166) 2023-07-19 13:05:53,203 DEBUG [RS:2;jenkins-hbase4:45863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:53,203 INFO [RS:2;jenkins-hbase4:45863] regionserver.LeaseManager(133): Closed leases 2023-07-19 13:05:53,203 INFO [RS:2;jenkins-hbase4:45863] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-19 13:05:53,203 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 13:05:53,204 INFO [RS:2;jenkins-hbase4:45863] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45863 2023-07-19 13:05:53,207 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45863,1689771933921 2023-07-19 13:05:53,207 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:53,207 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-19 13:05:53,208 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,45863,1689771933921] 2023-07-19 13:05:53,208 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,45863,1689771933921; numProcessing=1 2023-07-19 13:05:53,209 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,45863,1689771933921 already deleted, retry=false 2023-07-19 13:05:53,210 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,45863,1689771933921 expired; onlineServers=0 2023-07-19 13:05:53,210 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,40839,1689771927809' ***** 2023-07-19 13:05:53,210 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-19 13:05:53,210 DEBUG [M:0;jenkins-hbase4:40839] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c0fd7ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-19 13:05:53,210 INFO [M:0;jenkins-hbase4:40839] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,40839,1689771927809 2023-07-19 13:05:53,210 INFO [M:0;jenkins-hbase4:40839] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,40839,1689771927809; all regions closed. 2023-07-19 13:05:53,210 DEBUG [M:0;jenkins-hbase4:40839] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-19 13:05:53,210 DEBUG [M:0;jenkins-hbase4:40839] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-19 13:05:53,211 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-19 13:05:53,211 DEBUG [M:0;jenkins-hbase4:40839] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-19 13:05:53,211 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689771930217] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689771930217,5,FailOnTimeoutGroup] 2023-07-19 13:05:53,211 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689771930218] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689771930218,5,FailOnTimeoutGroup] 2023-07-19 13:05:53,211 INFO [M:0;jenkins-hbase4:40839] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-19 13:05:53,211 INFO [M:0;jenkins-hbase4:40839] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-19 13:05:53,211 INFO [M:0;jenkins-hbase4:40839] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-19 13:05:53,211 DEBUG [M:0;jenkins-hbase4:40839] master.HMaster(1512): Stopping service threads 2023-07-19 13:05:53,212 INFO [M:0;jenkins-hbase4:40839] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-19 13:05:53,212 ERROR [M:0;jenkins-hbase4:40839] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] 2023-07-19 13:05:53,212 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master Thread[IPC Client (1775694626) connection to localhost/127.0.0.1:33275 from jenkins,5,PEWorkerGroup] 2023-07-19 13:05:53,212 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-19 13:05:53,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-19 13:05:53,212 INFO [M:0;jenkins-hbase4:40839] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-19 13:05:53,213 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-19 13:05:53,213 DEBUG [M:0;jenkins-hbase4:40839] zookeeper.ZKUtil(398): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-19 13:05:53,213 WARN [M:0;jenkins-hbase4:40839] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-19 13:05:53,213 INFO [M:0;jenkins-hbase4:40839] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-19 13:05:53,213 INFO [M:0;jenkins-hbase4:40839] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-19 13:05:53,214 DEBUG [M:0;jenkins-hbase4:40839] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-19 13:05:53,214 INFO [M:0;jenkins-hbase4:40839] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 13:05:53,214 DEBUG [M:0;jenkins-hbase4:40839] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 13:05:53,214 DEBUG [M:0;jenkins-hbase4:40839] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-19 13:05:53,214 DEBUG [M:0;jenkins-hbase4:40839] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 13:05:53,214 INFO [M:0;jenkins-hbase4:40839] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=90.77 KB heapSize=109.49 KB 2023-07-19 13:05:53,234 INFO [M:0;jenkins-hbase4:40839] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=90.77 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ab94f3900e9e4fab9739236d9cf1fae0 2023-07-19 13:05:53,242 DEBUG [M:0;jenkins-hbase4:40839] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ab94f3900e9e4fab9739236d9cf1fae0 as hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ab94f3900e9e4fab9739236d9cf1fae0 2023-07-19 13:05:53,249 INFO [M:0;jenkins-hbase4:40839] regionserver.HStore(1080): Added hdfs://localhost:33275/user/jenkins/test-data/c3d2b5c2-9341-3eed-2fa8-e6942acd1df5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ab94f3900e9e4fab9739236d9cf1fae0, entries=27, sequenceid=212, filesize=8.2 K 2023-07-19 13:05:53,250 INFO [M:0;jenkins-hbase4:40839] regionserver.HRegion(2948): Finished flush of dataSize ~90.77 KB/92953, heapSize ~109.48 KB/112104, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 36ms, sequenceid=212, compaction requested=false 2023-07-19 13:05:53,251 INFO [M:0;jenkins-hbase4:40839] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-19 13:05:53,251 DEBUG [M:0;jenkins-hbase4:40839] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-19 13:05:53,257 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-19 13:05:53,257 INFO [M:0;jenkins-hbase4:40839] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-19 13:05:53,257 INFO [M:0;jenkins-hbase4:40839] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40839 2023-07-19 13:05:53,260 DEBUG [M:0;jenkins-hbase4:40839] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,40839,1689771927809 already deleted, retry=false 2023-07-19 13:05:53,467 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 13:05:53,467 INFO [M:0;jenkins-hbase4:40839] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,40839,1689771927809; zookeeper connection closed. 2023-07-19 13:05:53,467 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): master:40839-0x1017dafe8aa0000, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 13:05:53,567 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 13:05:53,567 INFO [RS:2;jenkins-hbase4:45863] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,45863,1689771933921; zookeeper connection closed. 2023-07-19 13:05:53,567 DEBUG [Listener at localhost/39077-EventThread] zookeeper.ZKWatcher(600): regionserver:45863-0x1017dafe8aa0008, quorum=127.0.0.1:58251, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-19 13:05:53,567 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@48b63c01] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@48b63c01 2023-07-19 13:05:53,567 INFO [Listener at localhost/39077] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-19 13:05:53,568 WARN [Listener at localhost/39077] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-19 13:05:53,571 INFO [Listener at localhost/39077] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 13:05:53,677 WARN [BP-1815553540-172.31.14.131-1689771924702 heartbeating to localhost/127.0.0.1:33275] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-19 13:05:53,677 WARN [BP-1815553540-172.31.14.131-1689771924702 heartbeating to localhost/127.0.0.1:33275] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1815553540-172.31.14.131-1689771924702 (Datanode Uuid 6fb50309-88be-4da3-adde-59fbc3578eed) service to localhost/127.0.0.1:33275 2023-07-19 13:05:53,679 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1/dfs/data/data3/current/BP-1815553540-172.31.14.131-1689771924702] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 13:05:53,679 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1/dfs/data/data4/current/BP-1815553540-172.31.14.131-1689771924702] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 13:05:53,680 WARN [Listener at localhost/39077] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-19 13:05:53,683 INFO [Listener at localhost/39077] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 13:05:53,726 WARN [BP-1815553540-172.31.14.131-1689771924702 heartbeating to localhost/127.0.0.1:33275] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1815553540-172.31.14.131-1689771924702 (Datanode Uuid 50b8eed1-fa26-49b1-8770-360fc1e33917) service to localhost/127.0.0.1:33275 2023-07-19 13:05:53,727 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1/dfs/data/data1/current/BP-1815553540-172.31.14.131-1689771924702] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 13:05:53,727 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0929219b-a3f5-e944-6628-5f85758ca43e/cluster_e881d367-5d63-c93c-2618-5c25e9e133e1/dfs/data/data2/current/BP-1815553540-172.31.14.131-1689771924702] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-19 13:05:53,830 INFO [Listener at localhost/39077] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-19 13:05:53,945 INFO [Listener at localhost/39077] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-19 13:05:53,982 INFO [Listener at localhost/39077] hbase.HBaseTestingUtility(1293): Minicluster is down